|
2
|
Ostropolets A, Albogami Y, Conover M, Banda JM, Baumgartner WA, Blacketer C, Desai P, DuVall SL, Fortin S, Gilbert JP, Golozar A, Ide J, Kanter AS, Kern DM, Kim C, Lai LYH, Li C, Liu F, Lynch KE, Minty E, Neves MI, Ng DQ, Obene T, Pera V, Pratt N, Rao G, Rappoport N, Reinecke I, Saroufim P, Shoaibi A, Simon K, Suchard MA, Swerdel JN, Voss EA, Weaver J, Zhang L, Hripcsak G, Ryan PB. Reproducible variability: assessing investigator discordance across 9 research teams attempting to reproduce the same observational study. J Am Med Inform Assoc 2023; 30:859-868. [PMID: 36826399 PMCID: PMC10114120 DOI: 10.1093/jamia/ocad009] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2022] [Revised: 01/04/2023] [Accepted: 01/23/2023] [Indexed: 02/25/2023] Open
Abstract
OBJECTIVE Observational studies can impact patient care but must be robust and reproducible. Nonreproducibility is primarily caused by unclear reporting of design choices and analytic procedures. This study aimed to: (1) assess how the study logic described in an observational study could be interpreted by independent researchers and (2) quantify the impact of interpretations' variability on patient characteristics. MATERIALS AND METHODS Nine teams of highly qualified researchers reproduced a cohort from a study by Albogami et al. The teams were provided the clinical codes and access to the tools to create cohort definitions such that the only variable part was their logic choices. We executed teams' cohort definitions against the database and compared the number of subjects, patient overlap, and patient characteristics. RESULTS On average, the teams' interpretations fully aligned with the master implementation in 4 out of 10 inclusion criteria with at least 4 deviations per team. Cohorts' size varied from one-third of the master cohort size to 10 times the cohort size (2159-63 619 subjects compared to 6196 subjects). Median agreement was 9.4% (interquartile range 15.3-16.2%). The teams' cohorts significantly differed from the master implementation by at least 2 baseline characteristics, and most of the teams differed by at least 5. CONCLUSIONS Independent research teams attempting to reproduce the study based on its free-text description alone produce different implementations that vary in the population size and composition. Sharing analytical code supported by a common data model and open-source tools allows reproducing a study unambiguously thereby preserving initial design choices.
Collapse
Affiliation(s)
- Anna Ostropolets
- Department of Biomedical Informatics, Columbia University Irving Medical Center, New York, New York, USA
| | - Yasser Albogami
- Department of Clinical Pharmacy, College of Pharmacy, King Saud University, Riyadh, Saudi Arabia
| | - Mitchell Conover
- Observational Health Data Analytics, Janssen Research & Development, Titusville, New Jersey, USA
| | - Juan M Banda
- Department of Computer Science, Georgia State University, Atlanta, Georgia, USA
| | - William A Baumgartner
- Division of General Internal Medicine, Department of Medicine, University of Colorado Anschutz Medical Campus, Aurora, Colorado, USA
| | - Clair Blacketer
- Observational Health Data Analytics, Janssen Research & Development, Titusville, New Jersey, USA
| | - Priyamvada Desai
- Research IT, Technology and Digital Solutions, Stanford Medicine, Stanford, California, USA
| | - Scott L DuVall
- VA Salt Lake City Health Care System, Salt Lake City, Utah, USA
- Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, Utah, USA
| | - Stephen Fortin
- Observational Health Data Analytics, Janssen Research & Development, Titusville, New Jersey, USA
| | - James P Gilbert
- Observational Health Data Analytics, Janssen Research & Development, Titusville, New Jersey, USA
| | | | - Joshua Ide
- Johnson & Johnson, Titusville, New Jersey, USA
| | - Andrew S Kanter
- Department of Biomedical Informatics, Columbia University Irving Medical Center, New York, New York, USA
| | - David M Kern
- Observational Health Data Analytics, Janssen Research & Development, Titusville, New Jersey, USA
| | - Chungsoo Kim
- Department of Biomedical Sciences, Ajou University Graduate School of Medicine, Suwon, South Korea
| | - Lana Y H Lai
- Department of Informatics, Imaging & Data Sciences, University of Manchester, Manchester, UK
| | - Chenyu Li
- Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
| | - Feifan Liu
- Department of Population and Quantitative Health Sciences, University of Massachusetts Chan Medical School, Worcester, Massachusetts, USA
| | - Kristine E Lynch
- VA Salt Lake City Health Care System, Salt Lake City, Utah, USA
- Department of Internal Medicine, University of Utah School of Medicine, Salt Lake City, Utah, USA
| | - Evan Minty
- O’Brien Institute for Public Health, Faculty of Medicine, University of Calgary, Calgary, Canada
| | | | - Ding Quan Ng
- Department of Pharmaceutical Sciences, School of Pharmacy & Pharmaceutical Sciences, University of California, Irvine, California, USA
| | - Tontel Obene
- Mississippi Urban Research Center, Jackson State University, Jackson, Mississippi, USA
| | - Victor Pera
- Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, The Netherlands
| | - Nicole Pratt
- Quality Use of Medicines and Pharmacy Research Centre, University of South Australia, Adelaide, Australia
| | - Gowtham Rao
- Observational Health Data Analytics, Janssen Research & Development, Titusville, New Jersey, USA
| | - Nadav Rappoport
- Department of Software and Information Systems Engineering, Ben-Gurion University of the Negev, Israel
| | - Ines Reinecke
- Institute for Medical Informatics and Biometry, Carl Gustav Carus Faculty of Medicine, Technische Universität Dresden, Dresden, Germany
| | - Paola Saroufim
- Cleveland Institute for Computational Biology, Case Western Reserve University, Cleveland, Ohio, USA
| | - Azza Shoaibi
- Observational Health Data Analytics, Janssen Research & Development, Titusville, New Jersey, USA
| | - Katherine Simon
- VA Tennessee Valley Health Care System, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Marc A Suchard
- Department of Biostatistics, University of California, Los Angeles, California, USA
- Department of Human Genetics, University of California, Los Angeles, California, USA
| | - Joel N Swerdel
- Observational Health Data Analytics, Janssen Research & Development, Titusville, New Jersey, USA
| | - Erica A Voss
- Observational Health Data Analytics, Janssen Research & Development, Titusville, New Jersey, USA
| | - James Weaver
- Observational Health Data Analytics, Janssen Research & Development, Titusville, New Jersey, USA
| | - Linying Zhang
- Department of Biomedical Informatics, Columbia University Irving Medical Center, New York, New York, USA
| | - George Hripcsak
- Department of Biomedical Informatics, Columbia University Irving Medical Center, New York, New York, USA
- Medical Informatics Services, New York-Presbyterian Hospital, New York, New York, USA
| | - Patrick B Ryan
- Department of Biomedical Informatics, Columbia University Irving Medical Center, New York, New York, USA
- Observational Health Data Analytics, Janssen Research & Development, Titusville, New Jersey, USA
| |
Collapse
|
|
3
|
Shokraneh F. Reproducibility and replicability of systematic reviews. World J Meta-Anal 2019; 7:66-71. [DOI: 10.13105/wjma.v7.i3.66] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/18/2019] [Revised: 03/23/2019] [Accepted: 03/26/2019] [Indexed: 02/06/2023] Open
Abstract
Irreproducibility of research causes a major concern in academia. This concern affects all study designs regardless of scientific fields. Without testing the reproducibility and replicability it is almost impossible to repeat the research and to gain the same or similar results. In addition, irreproducibility limits the translation of research findings into practice where the same results are expected. To find the solutions, the Interacademy Partnership for Health gathered academics from established networks of science, medicine and engineering around a table to introduce seven strategies that can enhance the reproducibility: pre-registration, open methods, open data, collaboration, automation, reporting guidelines, and post-publication reviews. The current editorial discusses the generalisability and practicality of these strategies to systematic reviews and claims that systematic reviews have even a greater potential than other research designs to lead the movement toward the reproducibility of research. Moreover, I discuss the potential of reproducibility, on the other hand, to upgrade the systematic review from review to research. Furthermore, there are references to the successful and ongoing practices from collaborative efforts around the world to encourage the systematic reviewers, the journal editors and publishers, the organizations linked to evidence synthesis, and the funders and policy makers to facilitate this movement and to gain the public trust in research.
Collapse
Affiliation(s)
- Farhad Shokraneh
- Division of Psychiatry and Applied Psychology, Institute of Mental Health, School of Medicine, University of Nottingham, Nottingham NG7 2TU, United Kingdom
| |
Collapse
|
|
4
|
Keltie K, Elneil S, Monga A, Patrick H, Powell J, Campbell B, Sims AJ. Complications following vaginal mesh procedures for stress urinary incontinence: an 8 year study of 92,246 women. Sci Rep 2017; 7:12015. [PMID: 28931856 PMCID: PMC5607307 DOI: 10.1038/s41598-017-11821-w] [Citation(s) in RCA: 107] [Impact Index Per Article: 13.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2017] [Accepted: 08/30/2017] [Indexed: 12/02/2022] Open
Abstract
Complications of surgical mesh procedures have led to legal cases against manufacturers worldwide and to national inquiries about their safety. The aim of this study was to investigate the rate of adverse events of these procedures for stress urinary incontinence in England over 8 years. This was a retrospective cohort study of first-time tension-free vaginal tape (TVT), trans-obturator tape (TOT) or suprapubic sling (SS) surgical mesh procedures between April 2007 and March 2015. Cases were identified from the Hospital Episode Statistics database. Outcomes included number and type of procedures, including those potentially confounded by concomitant procedures, and frequency, nature and timing of complications. 92,246 first-time surgical mesh procedures (56,648 TVT, 34,704 TOT, 834 SS and 60 combinations) were identified, including 68,002 unconfounded procedures. Peri-procedural and 30-day complication rates in the unconfounded cohort were 2.4 [2.3–2.5]% and 1.7 [1.6–1.8]% respectively; 5.9 [5.7–6.1]% were readmitted at least once within 5 years for further mesh intervention or symptoms of complications, the highest risk being within the first 2 years. Complication rates were higher in the potentially confounded cohort. The complication rate within 5 years of the mesh procedure was 9.8 [9.6:10.0]% This evidence can inform future decision-making on this procedure.
Collapse
Affiliation(s)
- Kim Keltie
- Newcastle upon Tyne Hospitals NHS Foundation Trust, Newcastle upon Tyne, UK.,Institute of Cellular Medicine, Faculty of Medical Sciences, University of Newcastle upon Tyne, Newcastle upon Tyne, UK
| | - Sohier Elneil
- University College London Hospitals NHS Foundation Trust, London, UK
| | - Ashwani Monga
- University Hospital of Southampton NHS Foundation Trust, Southampton, UK
| | - Hannah Patrick
- National Institute for Health and Care Excellence, London, UK
| | - John Powell
- National Institute for Health and Care Excellence, London, UK.,Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
| | - Bruce Campbell
- University of Exeter Medical School, University of Exeter, Exeter, UK
| | - Andrew J Sims
- Newcastle upon Tyne Hospitals NHS Foundation Trust, Newcastle upon Tyne, UK. .,Institute of Cellular Medicine, Faculty of Medical Sciences, University of Newcastle upon Tyne, Newcastle upon Tyne, UK.
| |
Collapse
|