Reproducibility and replicability of systematic reviews

Abstract

Irreproducibility of research causes a major concern in academia. This concern affects all study designs regardless of scientific fields. Without testing the reproducibility and replicability it is almost impossible to repeat the research and to gain the same or similar results. In addition, irreproducibility limits the translation of research findings into practice where the same results are expected. To find the solutions, the Interacademy Partnership for Health gathered academics from established networks of science, medicine and engineering around a table to introduce seven strategies that can enhance the reproducibility: pre-registration, open methods, open data, collaboration, automation, reporting guidelines, and post-publication reviews. The current editorial discusses the generalisability and practicality of these strategies to systematic reviews and claims that systematic reviews have even a greater potential than other research designs to lead the movement toward the reproducibility of research. Moreover, I discuss the potential of reproducibility, on the other hand, to upgrade the systematic review from review to research. Furthermore, there are references to the successful and ongoing practices from collaborative efforts around the world to encourage the systematic reviewers, the journal editors and publishers, the organizations linked to evidence synthesis, and the funders and policy makers to facilitate this movement and to gain the public trust in research.

Key Words: Systematic review; Meta-analysis; Reproducibility of results; Automation; Data science; Data anonymization; Datasets; Guideline adherence; Guideline; Peer-review

Core tip: Reproducibility increases the practicality of the research findings and gains the public trust in research. The ongoing developments in automation of systematic reviews, availability of pre-registration platform, dealing more with secondary data or anonymized primary data, the collaboration culture among the organizations who produce systematic reviews, and finally having an update step that mandates replicability are all reasons that systematic reviews have the potential to lead the movement toward the reproducibility among the other research designs. Meanwhile, reproducibility can help the systematic reviews to be considered as research design rather than literature review.

INTRODUCTION

Systematic reviews are at high levels of evidence hierarchy in clinical practice[1]. People who are involved in healthcare systems usually use systematic reviews in research, policy, and practice[3] trusting the reproducibility of the results when implemented[2]. At the same time, some criticize that the systematic reviews are literature reviews not research[4,5]. To utilize the systematic reviews in practice and to call them research studies, we need reproducibility testing; and to ensure that a systematic review is reproducible it is important to design, to record and to report systematic reviews in a transparent and reproducible way and to prioritize and fund reproducible reviews[6]. Some suggest that a team independent from the original team can repeat the systematic reviews to ensure the reproducibility[7]. Since conducting systematic reviews is already time-consuming[8] and resource-rating[9], it is arguable how adding more steps such as reproducibility test that requires more time and resources could reduce waste and increase value.

In context of this paper, reproducibility is re-conducting the same study, using the same methods and data by a different researcher or team and the replicability is re-doing the same study to gather new data or recollect the data[10].

To provide solutions for irreproducibility, the Interacademy Partnership for Health introduced seven strategy to enhance the reproducibility practice in science[11]. This editorial discusses the progress with using these strategies in systematic reviewing process and calls for collaboration in all levels of system to enhance the reproducibility of systematic reviews.

STRATEGY 1: PRE-REGISTRATION

Currently, prospective registration of systematic review protocols in PROSPERO, a register of systematic review protocols, is recommended[12]. Compared to clinical trials with at least 17 registries[13] there is only one register for systematic reviews; however, unlike clinical trials, it is not yet mandatory to register systematic reviews prospectively[14]. Today, PROSPERO covers only 30000 records of conducted, ongoing, awaiting, and abandoned review family (less than a third of 100000 systematic reviews in MEDLINE)[15], it does not support the quality control mechanism[16], and it lacks a rigor follow-up procedure for abandoned systematic reviews[17]. To look at the bright side, there is an association between registration of the published reviews and the quality of these reviews[18]. Allocating more resources to this register, training and encouraging the systematic reviewers to register their reviews, and making the pre-registration a standard for bias control will push the reproducibility theory toward practice.

STRATEGY 2: OPEN METHODS

Researchers should share search strategies for all databases[19] and analytical codes for meta-analysis[20] as part of the methods of systematic reviews. Following to the prospective registration and publication of the protocol, the researchers and the research audiences could assess the reproducibility and detect if any variation from the protocol could have important implementation messages for research, policy and practice[12]. This practice is not just to test the reproducibility but also to replicate another analysis or a new update for the systematic review. None of these are possible without access to all search strategies and statistical codes for meta-analysis.

STRATEGY 3: OPEN DATA

Search results (excluding copyrighted abstract and database-specific meta-data) in Research Information Systems (RIS) format[21] and extracted data and meta-data from the studies are the main resulting dataset during the systematic reviewing[22-24]. Access to open data from systematic reviews makes it possible to re-screen the search results, to de-duplicate the update searches, to re-run the meta-analyses, and to test the reproducibility of searching, screening, and data analysis steps. Besides, these data will have more value if they have been shared beside their associated meta-data following FAIR guidelines (findable, accessible, inter-operable, and reusable)[25]. There have already been calls for sharing the data from systematic reviews but there is no policy or action in place[22-24]. Sharing the data from all systematic reviews can lead into data-driven innovations with potential for knowledge discovery and saving the waste of resources and lives.

STRATEGY 4: COLLABORATION

Collaboration among research teams in small or large scale increases the chance for more expertise input and enhances the error detection and fixation practice[26,27]. Sharing the data among collaborators or interested research groups could bring together the data and resources for re-analyzing the same data[20] or innovations[23] that are impossible without such collaboration. It is not good practice to hold the data for years hoping to receive funding or innovating while sharing could result in faster innovation, receiving credits or collaboration in grant applications[26,27]. It also raises the morality and mortality question that is it ethical to hold the data when sharing it could lead to decisions that can save public resources and lives, and reduce the waste. The data extracted from other primary research for systematic reviews cannot be owned by the systematic reviewers or organizations that produce the systematic reviews.

STRATEGY 5: AUTOMATION

International Collaboration for the Automation of Systematic Reviews produces annual report of progress for automation of systematic reviews[28-30]. This collaboration seems to understand well that the automation is a key for reproducibility and follows Vienna Principles that also emphasize on the replicability of automation activities and sharing the program codes for wider use by the community[28]. The value of the automation becomes more obvious looking at reports of human errors in systematic reviews in searching[31] and data extraction steps[32]. The service provided by machine can speed the process and reduce the waste caused by human errors through standardization of practices such as statistical analysis or systematic review write-up steps[30,33]. Despite all technological development, systematic reviewers have underused the automation tools[34]. Currently, Systematic Review Data Repository[35], EPPI-Reviewer[36], Study-Based Registers[37], and Evidence Pipeline as semi-automated systems have the potential to evolve into automated systems for systematic reviews.

STRATEGY 6: REPORTING GUIDELINES

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)[38] now celebrates a decade of being used in reporting step of systematic reviews and major journals enforce the systematic reviewers to follow the PRISMA family guidelines in reporting. Such reporting guidelines are helping researchers to report certain items for publications and it is not their primary purpose to advocate the reproducibility[6]. There is an update of PRISMA 2019 in progress that will include more items and some these items can maximize the reproducibility practice[6].

STRATEGY 7: POST-PUBLICATION REVIEW

Pre-publication peer-reviews are limited to a few people while post-publication reviews give chance for wider audience to appraise and comment on some aspect of the research. Post-publication activities take many forms including letter to editor, commentary, blogs, and other social media posts[26]. These reviews are separate and independent from the original research and the only connection is through a link or citation. As a result, it is hardly possible to find all these reviews integrated in one place. This problem expands when there are retractions to the original systematic reviews or the findings are published in salami of papers. Such post-publication reviews, however, are encouraged in particular for systematic reviews because they can be taken into account in the next updates of the current systematic review. Having an update step in development of systematic reviews, unlike other published literature, is a unique advantage of systematic reviews allowing the reviewers to correct their mistakes and errors or to consider addition of new data or aspect to the review.

OPEN PROCESS: EMBEDDED REPRODUCIBILITY IN AGREEMENT CHECKS

As an addition to these strategies, it is also important not to overlook the process of the systematic reviewing and its connection to reproducibility. The routine practice in systematic reviews is to involve at least two researchers in screening and data extraction steps to reduce human errors[32,39] through double-checking of the decision and to reach an agreement. Such agreement sometimes requires a discussion between two reviewers or inviting the comments from another usually senior researcher. It means the decision on eligibility of studies or accuracy of data extraction is being replicated twice or three times. Since this process itself is replicating part of the review and has value for improving the reproducibility, some of the automation and semi-automation systems allow the researchers to document the process of double- and triple-checking within the system but for transparency purposes, this needs to be shared as well. In other words, the process should be documented and shared publicly.

SYSTEMATIC REVIEWS AS ROLE MODEL FOR OTHER RESEARCH DESIGNS

Systematic reviews have the great potential to lead the reproducibility practice among the rest of study designs in scientific fields because: A. Having an update step allows the systematic reviews to be corrected and helps in advancing ‘living systematic reviews’; B. Making a unique progress in automation of systematic reviews helps researchers to save time and resources in every step of systematic reviewing; C. Provision of protocol and methods facilitates the replication of systematic review in update step. To make such role model, the organizations whose main activity includes producing systematic reviews should come together and collaborate on developing policies on reproducibility and sharing the data and methods from within the systematic reviews. On the other hand, these organizations have their own journal platforms and the journal publishers themselves need to engage in this policy development as well. To avoid a meta-waste, Cochrane Database of Systematic Reviews, Systematic Reviews journal, World Journal of Meta-Analysis, JBI Database of Systematic Reviews and Implementation Reports, and Environmental Evidence now have a great opportunity to come together and set the bars on reproducibility of systematic reviews.