Network meta-analyses: Methodological prerequisites and clinical usefulness

Abstract

It is an undeniable fact that systematic reviews play a crucial role in informing clinical practice; however, conventional head-to-head meta-analyses do have limitations. In particular, studies can only be compared in a pair-wise fashion, and conclusions can only be drawn in the light of direct evidence. In contrast, network meta-analyses can not only compare multiple interventions but also utilize indirect evidence which increases their precision. On top of that, they can also rank competing interventions. In this mini-review, we have aimed to elaborate on the principles and techniques governing network meta-analyses to achieve a methodologically sound synthesis, thus enabling safe conclusions to be drawn in clinical practice. We have emphasized the prerequisites of a well-conducted Network Meta-Analysis (NMA), the value of selecting appropriate outcomes according to guidelines for transparent reporting, and the clarity achieved via sophisticated graphical tools. What is more, we have addressed the importance of incorporating the level of evidence into the results and interpreting the findings according to validated appraisal systems (i.e., the Grade of Recommendations, Assessment, Development, and Evaluation system - GRADE). Lastly, we have addressed the possibility of planning future research via NMAs. Thus, we can conclude that NMAs could be of great value to clinical practice.

Key Words: Network meta-analysis; Quality of evidence; Evidence-based medicine; Systematic reviews

Core Tip: Systematic reviews with or without meta-analyses provide the highest quality of evidence, thus lying on the top of evidence-based medicine hierarchy. However, pair-wise meta-analyses present the inherent limitation of exclusively comparing direct evidence. By contrast, Network Meta-Analyses (NMAs) also consider indirect evidence, thereby offering additional useful information. Conducting an NMA, however, has certain requirements such as assuming that transitivity across the included studies exists. What is more, maintaining sufficient statistical power in the analyses is crucial. In addition, performing head-to-head statistical comparisons before setting up networks of interventions is a prerequisite for a methodologically sound NMA, and selecting not only positive but also negative outcomes is required. Lastly, implementing quality appraisal systems to grade the level of evidence is highly recommended. Should all the above criteria be fulfilled, then accurate clinical conclusions can be drawn from an NMA.

INTRODUCTION

Due to the plethora of different interventions for various clinical entities[1] identifying the most efficient and safe treatment is among the prime interests of a researcher[2-4]. In the case of conventional meta-analyses, only two interventions can be compared at a time, and only those evaluated in head-to-head trials[5-7]. What is more, intervention effect estimates can only be calculated from direct evidence[2]. In contrast to pair-wise meta-analyses, network meta-analyses (NMA) enable not only simultaneous direct comparisons of multiple interventions but also indirect comparisons provided a common comparator is shared between interventions[2]. This is even possible in the case of two interventions that have never been directly compared[2]. In addition, interventions may also be ranked utilizing the surface under the cumulative ranking (SUCRA) curves, thus allowing for judgments such as which treatment presents the highest probability of being the most effective[2]. It is underlined that identifying more than one highly efficacious treatment in an NMA is a common phenomenon given the subtle differences in treatment rankings of the modalities lying on the top of ranking probabilitiy tables. Overall, incorporating the results from network meta-analyses into clinical practice guidelines could help clinicians select the best available intervention to improve healthcare.

PREREQUISITES FOR A WELL-CONDUCTED NMA - THE ASSUMPTION OF TRANSITIVITY AND HETEROGENEITY

For a systematic review of randomized evidence to qualify as a network meta-analysis, the assumption of transitivity must be fulfilled. To elaborate further, transitivity implies that it is possible to conclude hypothetical comparisons through a common comparator[6]. However, this is only possible in the absence of systematic differences between studies[8] with some degree of heterogeneity being permitted[6]. To illustrate further, heterogeneity is defined as a form of inter-study discrepancy due to differences that cause deviations in the observed effects other than sampling error[9]. However, when the discrepancy between studies exceeds that explained by clinical diversity, effects sizes cannot be safely estimated based on direct and indirect evidence and the distribution of effect modifiers needs to be examined[6].

PREREQUISITES FOR A WELL-CONDUCTED NMA- STATISTICAL POWER

It is worthy of note that the statistical power of a network of interventions should be sufficient to enable safe clinical conclusions to be drawn. To be more specific, the ratio between the number of included papers relative to the number of the competing interventions should be satisfactory. On top of that, the sample size per intervention arm as depicted by the size of the nodes in a network meta-analysis plot should also be robust enough (Figure 1). Lastly, prospective registration with systems such as the grade of recommendations, assessment, development, and evaluation system (GRADE) is valuable in assessing the heterogeneity and additional characteristics such as publication bias, indirectness, imprecision, the study limitations, and inconsistency[5].

Figure 1 A Network-Meta-Analysis plot example. Network meta-analysis plot including four competing interventions (i.e. A, B, C, and D). The nodes represent the included interventions with their size being proportional to sample size. The thickness of the edges connecting the nodes is reflected in the number of trials included in the given comparison. The edges depicted in green and yellow denote that the involved comparisons are at low and moderate risk of bias, respectively.

CONDUCTING PAIR-WISE META-ANALYSIS PRIOR TO NMA

Of additional note, for a given dataset, researchers must conduct not only NMA but also traditional pair-wise meta-analyses. To be more precise, one can take advantage of early exploration of the results of conventional pair-wise meta-analyses before setting up networks of interventions. Authors should then proceed with the network meta-analysis to take advantage of indirect evidence synthesis for them to supplement their study results.

PREREQUISITES FOR A WELL-CONDUCTED NMA- SELECTING APPROPRIATE OUTCOMES

In determining primary and secondary outcomes, both positive and negative results should be considered. Outcomes of primary interest should be prioritized over outcomes of secondary clinical importance to ensure that the findings will be clinically relevant. For instance, laboratory tests are not routinely considered as primary endpoints as they tend to not directly inform decisions. However, they may play an explanatory and/or adjuvant role in explaining the intervention outcome[10].

FOLLOWING GUIDELINES FOR TRANSPARENT REPORTING

The PRISMA guidelines represent a checklist of 27 items that may be used when reporting a systematic review of health interventions with or without meta-anlysis[11]. Hutton et al[12], in 2015, has expanded the original list by including 5 additional items that apply to network meta-analyses. Firstly, the geometry and summary of the intervention networks have been incorporated in the methods, including a diagrammatic representation and a brief description. What is more, the findings of inconsistency assessment can be included in addition to the presentation of the networks’ structure.

It should be noted that prospective registration (e.g. with PROSPERO database) of all NMAs is encouraged. By doing so, transparency is promoted and bias is prevented by avoiding unintended duplicate reviews[13]. It is also highlighted that adherence to a pre-existing protocol plays a crucial role in preventing selective outcome reporting[10,14,15]. In other words, registration of a systematic review in advance of study commencement precludes data manipulation and/or unethical reporting. Last but not least, prospective registration may enable researchers to assess whether the topic they intend to investigate has already been addressed by earlier authors, thus avoiding unnecessary research repetition.

SOPHISTICATED GRAPHICAL TOOLS IN NMA - DO WE NEED THEM?

Despite NMAs gaining popularity, a lot of criticism exists given their complex methodology discouraging clinicians from getting involved in this type of research[16]. This is due to the increased level of statistical and computational knowledge required. To tackle this issue, introducing graphical tools into the manuscript results in a significant increase in clarity and reproducibility[16].

What is more, competing interventions can be ranked from the most to the least effective via the use of SUCRA curves[2]. On the other hand, league tables enable a structured presentation of the result of each pair of comparisons with its corresponding 95% confidence intervals (Figure 2).

Figure 2 Hypothetical league table demonstrating standardized mean differences, from a network meta-analysis of five competing interventions, that is A-E. Statistically significant values are depicted in bold.

PUBLICATION BIAS IN NMA AND ITS IMPACT ON CLINICAL ESTIMATES

It has been evidenced that detection of publication bias (that is typically reporting positive more often than negative results) in NMA is not uncommon. Inevitably, introducing this kind of bias in meta-analysis threatens the validity of the results of the study as an “overly rosy picture” may be painted. To elaborate further, the evaluation of small study effects acts as a proxy for the assessment of publication bias. For the above assessment, a sophisticated statistical tool namely a comparison-adjusted funnel plot can be implemented. Apart from funnel plots, researchers can also employ Egger’s test to statistically evaluate the presence of small-study effects[16-18].

QUALITY APPRAISAL SYSTEMS AND TRANSITION TO CLINICAL PRACTICE

The GRADE system features 6 components[5], that are study limitations, heterogeneity, inconsistency, indirectness, imprecision, and publication bias[5,19]. The quality of evidence may be high, moderate, low, or very low. As a rule of thumb, randomized trials yield high-quality evidence, whereas observational studies more often than not offer a low quality of evidence with the risk of bias potentially affecting clinical judgment[20].

Potential limitations of randomized trials include failure to conceal allocation, failure to blind, loss to follow-up, and failure to appropriately consider the intention[20]. Guyatt et al[20], in 2011, also mentioned terminating a study early for apparent benefit, and selective reporting of outcomes according to the results. The indirectness may be due to patients deviating from those of interest, when the treatments have not been compared in head-to-head trials, and when there are different outcomes from those being expected from the study[21]. Furthermore, the contributions of biological and social factors to the magnitude of effect in the outcomes represents indirectness[21]. On the other hand, inconsistency is defined as a disagreement between direct and indirect evidence in NMA[19]. In addition, Salanti et al[19], have suggested the adoption of a quantitative approach to assess the risk of bias.

INVESTIGATING CLINICAL DIVERSITY IN NMA

It is an undeniable fact that a great many confounding factors can be encountered in a broad systematic review of randomized trials. Thus, conducting sensitivity analysis to delineate the impact of clinical heterogeneity factors is strongly recommended. For instance, the effect of low-quality trials, variation in intervention characteristics as well as differences due to variable outcome measurement tools needs to be considered in those secondary analyses. From a technical point of view, the researcher needs to improve the trial(s) with the above characteristics from the analysis, repeat the statistical tests and subsequently compare the new results with the findings of the primary analysis[22].

PLANNING FUTURE RESEARCH WITH NMA- IS IT POSSIBLE?

Directing the design of future studies based on NMA results appears to be of significant importance as mismanagement of resources can be overcome[23-25]. For a researcher to provide an estimate of whether the results of a subsequent trial are likely to change in the future, an interval plot should be considered. By visually inspecting an interval plot, an investigator can enable predictions on the efficacy of a particular intervention in a future trial[16,26,27].

IMPROVING INTERPRETATION OF NMA FINDINGS

To improve interpretability and clarity of the results of an NMA, researchers are encouraged to back-transform their data in a manner that interpretation of their results is improved. For instance, when it comes to Patient-Reported Outcome Measures, investigators can back-transform Standard Mean Differences to Mean Differences and subsequently assess their findings against the established minimal clinically important difference for a particular questionnaire[28].

CONCLUSION

Overall, NMAs play a crucial role in the decision-making process. As long as common methodological mistakes are avoided, researchers can produce reliable and accurate clinical conclusions.