Development of the Documentation and Appraisal Review Tool for systematic reviews

Table 1 Assessment of existing systematic review quality assessment tools

Need	Modified OQAQ	AMSTAR
Standardized quality assessment process across multiple reviewers with varying levels of experience	Insufficient detail to evaluate disputes	Confusing questions leading to inconsistent responses by same reviewer as well as between reviewers
Single tool to assess a variety of included research designs including randomized trials and observational studies	Insufficient detail on methods	Insufficient detail on methods
Detailed record of the review to facilitate updates of the evidence review	Insufficient detail for replication	Confusing questions leading to inconsistent responses by same reviewer and insufficient detail for replication
Training tool for junior epidemiologists and interns in systematic review methods	Insufficient detail on methods	Insufficient detail on methods

OQAQ: Overview Quality Assessment Questionnaire; AMSTAR: Assessment of Multiple Systematic Reviews.

Table 2 Documentation and Appraisal Review Tool for systematic reviews

Title of Systematic Review:
Author:
Publication date:		Article tracking number:
Reviewer:		Date completed:
1 Did the authors develop the research question(s) and inclusion/exclusion criteria before conducting the review?				Use this space to document the rationale for your answer
a	It was clear the authors developed the research question(s) and inclusion criteria before conducting the review and that they stated the question(s) clearly	Yes
b	Not described or cannot tell	No
2 Did the authors describe the search methods used to find evidence (original research) on the primary question(s)?				Use this space to document the rationale for your answer
a	Key words and/or MESH terms were stated and where feasible the search strategy was provided	Yes
b	Not described or cannot tell	No
3 Was the search for the evidence reasonably comprehensive? Were the following included?				Use this space to document the rationale for your answer
a	Search included at least two electronic sources	Yes	No
b	Authors chose the most applicable electronic databases (e.g., CINAHL for nursing journals, EMBASE for pharmaceutical journals, and MEDLINE for general, comprehensive search) and only limited search by date when performing an update of a previous systematic review	Yes	No
c	Search methods are likely to capture all relevant studies (e.g., includes languages other than English; gray literature such as conference proceedings, dissertations, theses, clinical trials registries and other reports) and authors hand-searched journals or reference lists to identify published studies which were not electronically available	Yes	No
4 Did the authors do the following when selecting studies for the review?				Use this space to document the rationale for your answer
a	Provide in the inclusion criteria: population, intervention, outcome and study design?	Yes	No
b	State whether the selection criteria were applied independently by more than one person?	Yes	No
c	State how disagreements were resolved during study selection?	Yes	No
d	Provide a flowchart or descriptive summary of the included and excluded studies?	Yes	No
e	Include all study designs appropriate for the research questions posed?	Yes	No
5 Were the characteristics of the included studies provided? (in an aggregated form such as a table, data from the original studies were provided on the participants, interventions and outcomes)				Use this space to document the rationale for your answer
a	Yes
b	Partially
c	No
6 Did the authors make any statements about assessing for publication bias?				Use this space to document the rationale for your answer
a	The authors did assess for publication bias and if publication bias was detected they stated how it was handled	Yes
b	The authors did assess for publication bias but did not state how it was handled if it was detected	Partially
c	Not described or cannot tell	No
7 Did the authors do the following to assess the overall quality of the individual studies included in the review?				Use this space to document the rationale for your answer
a	Was the quality assessment specified with adequate detail to permit replication?	Yes	No
b	Was the quality assessment conducted independently by more than one person?	Yes	No
c	Did the authors state how disagreements were resolved during the quality assessment?	Yes	No
8 Did the authors appropriately assess for quality by appropriately examining the following sources of bias in all of the included studies?				Use this space to document the rationale for your answer
All studies:
a	Confounding (assessed comparability of study groups at start of study, was randomization successful?)	Yes	No
b	Sufficient sample size (only applicable to studies that summarize their results in a qualitative manner; it's not a concern for pooled results)	Yes	No
c	Outcome reporting bias (assessed for each outcome reported using a system such as the ORBIT classification system)	Yes	No
d	Follow up (assessed for completeness and any differential loss to follow-up)	Yes	No
For Randomized Controlled Trials only:
e	Randomization	Yes	No
f	Allocation concealment	Yes	No
g	Blinding	Yes	No
For Case-Control and Cohort Studies only:
h	Selection bias	Yes	No
i	Information bias--recall and completeness to follow-up	Yes	No
For Quasi-Experimental Studies only:
j	Differences between the first and second study measurement point - such as changes or improvements in other interventions, changes in measurement techniques or definitions, or aging of subjects	Yes	No
k	Selection bias	Yes	No
For Diagnostic Accuracy Studies only:
l	Selection (spectrum) bias - were subjects selected to be representative of patients to whom the test will be applied in clinical practice, and to represent the broadest spectrum of disease?	Yes	No
m	Verification bias - were all patients subjected to the same reference standard of diagnosis, and was it measured blindly and independently of the test?	Yes	No
9 Did the authors use appropriate methods to extract data from the included studies?				Use this space to document the rationale for your answer
a	Were standard forms developed and piloted prior to the systematic review conduct?	Yes	No
b	Did the authors ensure that data from the same study but that appeared in multiple publications were counted only once in the synthesis?	Yes	No
c	Was data extraction performed by more than one person?	Yes	No
10 Did the authors assess and account for heterogeneity (differences in participants, interventions, outcomes, trial design, quality or treatment effects) among the studies selected for the review?				Use this space to document the rationale for your answer
a	The authors stated the differences among the studies and how they accounted for those differences	Yes
b	The authors stated the differences but not how they accounted for them	Partially
c	Not described or cannot tell	No
11 Did the authors describe the methods they used to combine/synthesize the results of the relevant studies (to reach a conclusion) and were the methods used appropriate for the review question(s)?				Use this space to document the rationale for your answer
a	Methods were reported clearly enough to allow for replication. The overview included some assessment of the qualitative and quantitative heterogeneity of the study results and the results were appropriately combined/synthesized. For meta-analyses, an accepted pooling method (i.e., more than simple addition) was used. Or the authors state that the evidence is conflicting and that they can't combine/synthesize the results	Yes
b	The methods were reported clearly enough to allow for replication but they were not combined appropriately	Partially
c	Not described or cannot tell	No
12 Did the authors perform sensitivity analyses on any changes in protocol, assumptions, and study selection? (For example, using sensitivity analysis to compare results from fixed effects and random effects models)				Use this space to document the rationale for your answer
a	Sensitivity analyses were used when appropriate on all changes in a priori design	Yes
b	Sensitivity analyses were only used on some changes in a priori design	Partially
c	Not described or cannot tell	No
13 Are the conclusions of the authors supported by the reported data with consideration of the overall quality of that data?				Use this space to document the rationale for your answer
a	The conclusions are supported by the reported data and reflect both the scientific quality of the studies and the risk of bias in the data obtained from those studies	Yes
b	The authors failed to consider study quality and/or their conclusions were not supported by the data, or cannot tell	No
14 Were conflicts of interest stated and were individuals excluded from the review if they reported substantial financial and intellectual COIs?				Use this space to document the rationale for your answer
a	COIs were reported for each team member and individuals were excluded if they had substantial COIs	Yes
b	COIs were reported but it was not clear whether individuals were excluded based on their COIs	Partially
c	COIs were not reported and individuals were not excluded based on their COIs	No
15 On a scale of 1-10, how would you judge the overall quality of the paper?
Rating	Overall Comments
Good (8-10)
Fair (5-7)
Poor (< 5)

COIs: Conflicts of interests.

Table 3 Comparison of Documentation and Appraisal Review Tool to modified Overview Quality Assessment Questionnaire and Assessment of Multiple Systematic Reviews

DART questions	Corresponding AMSTAR question(s)	Corresponding modified OQAQ question(s)
(1) Did the authors develop the research question(s) and inclusion/exclusion criteria before conducting the review?	(1) Was an "a priori" design provided?	Not addressed
(2) Did the authors describe the search methods used to find evidence (original research) on the primary question(s)?	(3) Was a comprehensive literature search performed?	(1) Were the search methods used to find evidence on the primary question stated?
(2a) Are key words and/or MESH terms stated?	(3) Was a comprehensive literature search performed?	Not addressed
(3) Was the search for the evidence reasonably comprehensive?	(3) Was a comprehensive literature search performed?	(2) Was the search for evidence reasonably comprehensive?
(3a) Does the search include at least 2 databases?	(3) Was a comprehensive literature search performed?	Not addressed
(3b) Did the authors choose the most applicable electronic databases and only limit the search by date when performing an update?	Not addressed	Not addressed
(3c) Are search methods likely to capture all relevant studies and did the authors hand-search journals or reference lists to identify published studies which were not electronically available?	(3) Was a comprehensive literature search performed?	Not addressed
	(4) Was the status of publication (i.e., grey literature) used as an inclusion criterion?
(4a) Did the authors provide in the inclusion criteria: Population, intervention, outcome, and study design, when selecting studies for the review?	Not addressed	Not addressed
(4b) Did the authors state whether the selection criteria were applied by more than one person?1	(2) Was there duplicate study selection and data extraction?1	Not addressed
(4c) Did the authors state how disagreements were resolved during study selection?1	(2) Was there duplicate study selection and data extraction?1	Not addressed
(4d) Did the authors provide a flowchart or descriptive summary of the included and excluded studies?	(5) Was a list of studies (included and excluded) provided?	Not addressed
(4e) Did the authors include all study designs appropriate for the research questions posed?	Not addressed	Not addressed
(5) Were the characteristics of the included studies provided? (in an aggregated form such as a table, data from the original studies were provided on the participants, interventions and outcomes)	(6) Were the characteristics of the included studies provided?	Not addressed
(6) Did the authors make any statements about assessing for publication bias?	(10) Was the likelihood of publication bias assessed?	Not addressed
(7a) Was the quality assessment specified with adequate detail to permit replication?	(7) Was the scientific quality of the included studies assessed and documented?	(5) Were the criteria used for assessing the validity of the included studies reported?
(7b) Was the quality assessment conducted independently by more than one person?	Not addressed	Not addressed
(7c) Did the authors state how disagreements were resolved during the quality assessment?	Not addressed	Not addressed
(8) Did the authors appropriately assess for quality by appropriately examining the following sources of bias in all of the included studies: confounding, sufficient sample size, outcome reporting bias, follow-up, randomization, allocation concealment, blinding, selection bias, information bias, verification bias, and differences between the first and second study measurement point?	(7) Was the scientific quality of the included studies assessed and documented? (partial match)	(6) Was the validity of all studies referred to in the text assessed using appropriate criteria? (partial match)
(9) Did the authors use appropriate methods to extract data from the included studies?	Not addressed	Not addressed
(9a) Were standard forms developed and piloted prior to the systematic review conduct?	Not addressed	Not addressed
(9b) Did the authors ensure that data from the same study that appeared in multiple publications were counted only once in the synthesis?	Not addressed	Not addressed
(9c) Was data extraction performed by more than one person?	(2) Was there duplicate study selection and data extraction?	Not addressed
(10) Did the authors assess and account for heterogeneity (differences in participants, interventions, outcomes, and trial design, quality or treatment effects) among the studies selected for the review?	(9) Were the methods used to combine the findings of studies appropriate?	(7) Were the methods used to combine the findings of the relevant studies reported?
		(8) Were the findings of the relevant studies combined appropriately?
(11) Did the authors describe the methods they used to combine/synthesize the results of the relevant studies (to reach a conclusion) and were the methods used appropriate for the review question(s)?	(9) Were the methods used to combine the findings of studies appropriate?	(7) Were the methods used to combine the findings of the relevant studies reported?
		(8) Were the findings of the relevant studies combined appropriately?
(12) Did the authors perform sensitivity analyses on any changes in protocol, assumptions, and study selection? (For example, using sensitivity analysis to compare results from fixed effects and random effects models)	Not addressed	Not addressed
(13) Are the conclusions of the authors supported by the reported data with consideration of the overall quality of that data?	(8) Was the scientific quality of the included studies used appropriately in formulating conclusions? (partial match)	(9) Were the conclusions made by the author(s) supported by the data reported? (partial match)
(14) Were conflicts of interest stated and were individuals excluded from the review if they reported substantial financial and intellectual COIs?	(11) Was the conflict of interest stated? (partial match)	Not addressed
(15) On a scale of 1-10, how would you judge the overall quality of the paper?	Not addressed	(10) Overall quality

¹Separate questions in DART, but concepts not separated in AMSTAR. DART: Documentation and Appraisal Review Tool; OQAQ: Overview Quality Assessment Questionnaire; AMSTAR: Assessment of Multiple Systematic Reviews; COIs: Conflicts of interests.