Accuracy of artificial intelligence in meta-analysis: A comparative study of ChatGPT 4.0 and traditional methods in data synthesis

Abstract

BACKGROUND

Meta-analysis is a critical tool in evidence-based medicine, particularly in cardiology, where it synthesizes data from multiple studies to inform clinical decisions. This study explored the potential of using ChatGPT to streamline and enhance the meta-analysis process.

AIM

To investigate the potential of ChatGPT to conduct meta-analyses in interventional cardiology by comparing the results of ChatGPT-generated analyses with those of randomly selected, human-conducted meta-analyses on the same topic.

METHODS

We systematically searched PubMed for meta-analyses on interventional cardiology published in 2024. Five meta-analyses were randomly chosen. ChatGPT 4.0 was used to perform meta-analyses on the extracted data. We compared the results from ChatGPT with the original meta-analyses, focusing on key effect sizes, such as risk ratios (RR), hazard ratios, and odds ratios, along with their confidence intervals (CI) and P values.

RESULTS

The ChatGPT results showed high concordance with those of the original meta-analyses. For most outcomes, the effect measures and P values generated by ChatGPT closely matched those of the original studies, except for the RR of stent thrombosis in the Sreenivasan et al study, where ChatGPT reported a non-significant effect size, while the original study found it to be statistically significant. While minor discrepancies were observed in specific CI and P values, these differences did not alter the overall conclusions drawn from the analyses.

CONCLUSION

Our findings suggest the potential of ChatGPT in conducting meta-analyses in interventional cardiology. However, further research is needed to address the limitations of transparency and potential data quality issues, ensuring that AI-generated analyses are robust and trustworthy for clinical decision-making.

Key Words: ChatGPT; Artificial intelligence; Large language model; Meta-analysis; Statistical analysis; Methodology; Cardiology

Core Tip: This study explored the potential of using ChatGPT 4.0 to conduct meta-analyses. Five meta-analyses were systematically selected. ChatGPT 4.0 was used to perform meta-analyses on the extracted data, and the results were compared with the original meta-analyses. The results generated by ChatGPT 4.0 showed high concordance with those of the original meta-analyses. For most outcomes, the effect measures and p-values generated by ChatGPT 4.0 closely matched those of the original studies. Our findings suggest the potential of ChatGPT 4.0 in conducting meta-analyses in cardiology.

INTRODUCTION

Meta-analysis is a powerful tool in evidence-based medicine that provides a systematic approach for synthesizing data from multiple studies to derive conclusions about medical interventions. In interventional cardiology, where new technologies and procedures are constantly emerging, meta-analysis plays a vital role in evaluating their effectiveness and safety compared to existing methods. This information is crucial for informing clinical guidelines and practices[1]. The rigorous process of conducting a meta-analysis involves comprehensive literature searches, data extraction, statistical analysis, and critical interpretation, which often requires substantial expertise and time[2]. However, the advent of artificial intelligence (AI) technologies, such as ChatGPT developed by OpenAI, offers new opportunities to streamline and potentially expedite this process.

ChatGPT is an AI language model that uses machine learning techniques to generate human-like text based on the input it receives. It is trained on a vast text corpus, allowing it to understand and respond to various topics, including medical research[3]. Recent studies have shown that AI can effectively assist in various aspects of medical research[4]. For instance, AI has been used to automate the screening of abstracts and full-text articles during the systematic review process, thereby significantly reducing the time required for these tasks[5]. Moreover, AI-driven tools have demonstrated high accuracy in identifying relevant studies and extracting key data points, which are critical steps in conducting meta-analyses[6].

Despite these promising developments, the application of AI in meta-analyses remains in its early stages and there are several challenges to address. One of the primary concerns is the transparency and interpretability of AI-generated results. Although AI models such as ChatGPT can produce coherent and relevant text, the underlying decision-making process is often a “black box,” making it difficult for researchers to understand how conclusions are derived[7]. Ensuring that AI-generated analyses are transparent and can be validated by human experts is essential for their acceptance and use in clinical research[8]. Another challenge is the need for high-quality curated datasets to train the AI models. The reliability of AI-generated analyses depends on the quality of the input data, and any bias or error in the training data can affect the outcome[9]. Therefore, ongoing efforts to improve the data quality and develop robust training datasets are crucial for advancing the use of AI in meta-analyses.

This study investigated the potential of ChatGPT to conduct meta-analyses in interventional cardiology by comparing the results of ChatGPT-generated analyses with those of randomly selected, human-conducted meta-analyses on the same topic. By identifying potential discrepancies and insights, this study aimed to evaluate the reliability and utility of ChatGPT as a tool for supporting meta-analyses in this field.

MATERIALS AND METHODS

Search strategy and eligibility criteria

PubMed was systematically searched using the following search strategy: (interventional cardiology) OR (cardiac intervention) OR (catheterization) OR (cardiology) OR (cardiac catheterization) AND (angioplasty) OR (stenting) OR (percutaneous coronary intervention) OR (valvuloplasty) OR (transcatheter aortic valve replacement)) OR (TAVR) OR (angioplasty, balloon) OR (atherectomy, coronary) OR (valvuloplasty) AND (coronary artery disease) OR (myocardial infarction)) OR (ischemic heart disease)) OR (structural heart disease)) OR (congenital heart disease) OR (heart defects, congenital) AND (meta-analysis) OR (meta-analysis).

Two independent reviewers screened the retrieved titles and abstracts according to pre-established eligibility criteria, focusing on studies involving adult patients undergoing interventional cardiology procedures, investigating procedural effects or techniques, and reporting relevant clinical outcomes such as mortality, major adverse cardiovascular events (MACE), stent thrombosis, and revascularization rates. Only studies identified as meta-analyses that were published in peer-reviewed journals in 2024 were considered. Observational studies, randomized controlled trials, network meta-analyses, reviews, and articles that were not available in English were excluded.

Study selection

We chose five meta-analyses in our study because we believe this number provides a well-balanced mixture of different effect measures, software utilized by the included studies, and journals, offering a thorough insight into the capabilities of ChatGPT in statistical analyses. Five meta-analyses were randomly selected from the final list of eligible studies using a random number generator to minimize selection bias. Random selection ensured that each eligible study had an equal chance of being included, thereby avoiding subjective bias in article selection.

Data extraction from meta-analyses

Two independent reviewers (Tariq MD, Ahsan A) extracted quantitative data (events/total or events/non-events in the interventional and control groups) from each selected meta-analysis using a predefined data extraction sheet. Extracted data included study ID, journal in which the study was published, outcome measures, type of effect measure, and the effect measure value and p-value reported by the meta-analysis and generated by ChatGPT.

Data analysis by ChatGPT

The quantitative data extracted from the meta-analyses were inputted into ChatGPT 4.0 for analysis generation. ChatGPT was used to perform a meta-analysis to calculate the pooled effect measures [risk ratio (RR), hazard ratio (HR), and odds ratio (OR)] along with their corresponding 95% confidence intervals (CIs) and P values. The ChatGPT was instructed to utilize either a fixed-effects or random-effects model based on the specific approach employed in the original meta-analysis for consistency. The results generated by the ChatGPT were compared with the analysis results from each meta-analysis to identify any significant discrepancies. The prompts given to ChatGPT and the corresponding generated results are provided in the Supplementary material.

RESULTS

The initial search yielded a total of 135 results. Of these, five studies were randomly selected for inclusion in the comparative analysis[10-14]. All outcomes reported in each study were analyzed and compared with the results obtained using the ChatGPT. The included studies were published in the following journals: Journal of the American Heart Association[10], Circulation[11], American Journal of Cardiology[12], European Heart Journal[13], and International Journal of Cardiology[14]. The reported effect measures included RR, HR, and OR with their corresponding 95%CI and P values. A comparison of the effect measures and p-values between the original studies and the ChatGPT is summarized in Table 1. The comparison between the two groups for various outcome measures, including effect sizes and their 95%CIs, is depicted using bar graphs in Figure 1.

Figure 1  Comparison between the traditional and ChatGPT models for various outcome measures, including effect sizes and their 95% confidence interval.

Table 1 Comparison of effect measures and P values between original studies and ChatGPT.

Ref.	Journal	Software used for analysis	Outcome measures	Type of effect measure	Effect measure (original)	P value (original)	Effect measure (ChatGPT)	P value (ChatGPT)
Sreenivasan et al[10]	Journal of American Heart Association	R studio	MACE	Risk ratio (95%CI)	0.67 (0.55 -0.82)	< 0.001	0.67 (0.56-0.81)	< 0.001
			Cardiac death	Risk ratio (95%CI)	0.49 (0.34 -0.71)	< 0.001	0.46 (0.29-0.72)	< 0.001
			All-cause death	Risk ratio (95%CI)	0.81 (0.61 -1.07)	0.135	0.80 (0.60-1.06)	0.124
			Myocardial infarction	Risk ratio (95%CI)	0.82 (0.62-1.07)	0.144	0.84 (0.64-1.10)	0.197
			Target vessel MI	Risk ratio (95%CI)	0.61 (0.42-0.89)	0.011	0.61 (0.42-0.90)	0.011
			Target lesion revascularization	Risk ratio (95%CI)	0.67 (0.49-0.91)	0.011	0.66 (0.48-0.91)	0.010
			Target vessel revascularization	Risk ratio (95%CI)	0.60 (0.45-0.80)	< 0.001	0.60 (0.45-0.80)	< 0.001
			Stent thrombosis	Risk ratio (95%CI)	0.63 (0.40-0.99)	0.046	0.67 (0.42-1.06)	0.090
Patel et al[11]	Circulation	R studio	MACE	Hazard ratio (95%CI)	0.91 (0.87-0.96)	< 0.0001	0.94 (0.90-0.98)	0.005
			Cardiovascular death	Hazard ratio (95%CI)	0.86 (0.81-0.92)	< 0.0001	0.88 (0.81-0.96)	0.0037
			Myocardial infarction	Hazard ratio (95%CI)	0.95 (0.87-1.04)	0.29	0.97 (0.90-1.05)	0.429
			Stroke	Hazard ratio (95%CI)	0.99 (0.91-1.07)	0.77	0.87 (0.65-1.18)	0.380
Sacoransky et al[12]	American Journal of Cardiology	RevMan	LV thrombus formation	Odds ratio (95%CI)	0.28 (0.11-0.73)	< 0.01	0.37 (0.16-0.86)	0.021
			Major bleeding events	Odds ratio (95%CI)	2.85 (1.13-7.24)	0.03	2.82 (1.30-6.13)	0.009
			Systemic embolism	Odds ratio (95%CI)	0.37 (0.12-1.13)	0.08	0.42 (0.13-1.29)	0.130
Paolucci et al[13]	European Heart Journal	RevMan	Relationship between PMI (UDMI and UDMI-like definitions) and mortality	Hazard ratio (95%CI)	1.61(1.32- 1.97)	< 0.00001	1.61 (1.32-1.97)	< 0.00001
			Relationship between PMI (ARC-2 definition) and mortality	Hazard ratio (95%CI)	2.07 (1.40-3.08)	0.0003	2.07 (1.40-3.08)	0.0003
			Relationship between PMI (SCAI and SCAI-like definitions) and mortality	Hazard ratio (95%CI)	3.24 (2.36-4.44)	< 0.00001	3.24 (2.36-4.44)	< 0.00001
Ang et al[14]	International Journal of Surgery	Stata	Major vascular complications	Risk ratio (95%CI)	2.32 (1.73-3.11)	< 0.001	2.30 (1.72-3.09)	< 0.001
			Major bleeding	Risk ratio (95%CI)	1.61 (1.27-2.05)	< 0.001	1.61 (1.27-2.05)	< 0.001
			Aortic annulus rupture	Risk ratio (95%CI)	4.66 (1.67-13.01)	< 0.001	4.66 (1.67-13.01)	0.0033
			Cardiac tamponade	Risk ratio (95%CI)	3.0 (1.31-6.89)	0.01	3.0 (1.31-6.89)	0.0095
			Minor vascular complications	Risk ratio (95%CI)	1.43 (1.00-2.04)	0.05	1.43 (1.00-2.04)	0.052
			Stroke	Risk ratio (95%CI)	1.37 (0.81-2.32)	0.24	1.41 (0.82-2.40)	0.210
			In-hospital mortality	Risk ratio (95%CI)	1.86 (0.74-4.70)	0.19	1.86 (0.74-4.70)	0.187
			30 days mortality	Risk ratio (95%CI)	1.14 (0.532.46)	0.74	1.22 (0.55-2.71)	0.628
			Pacemaker implantation	Risk ratio (95%CI)	0.91 (0.70-1.18)	0.47	0.98 (0.75-1.26)	0.848

MACE: Major adverse cardiac events; MI: Myocardial infarction; LV: Left ventricular.

Study 1

Major adverse cardiovascular events: The RR calculated by ChatGPT (0.67, 95%CI: 0.56-0.81) was very close to the original (0.67, 95%CI: 0.55-0.82)[10], demonstrating high concordance. Both analyses indicated a statistically significant reduction in MACE, with P-values in agreement (P < 0.001 for both ChatGPT and the original).

Cardiac death: ChatGPT provided an RR of 0.46 (95%CI: 0.29-0.72) compared to the original 0.49 (95%CI: 0.34-0.71)[10]. Both analyses showed a significant reduction in cardiac death, with P < 0.001 for both.

All-cause death: The RR for ChatGPT (0.80, 95%CI: 0.60-1.06) was comparable to the original (0.81, 95%CI: 0.61-1.07)[10]. Neither analysis showed a statistically significant reduction in all-cause mortality (P = 0.124 for ChatGPT and 0.135 for original), suggesting close agreement.

Myocardial infarction: Both analyses yielded similar insignificant RRs, with ChatGPT reporting 0.84 (95%CI: 0.64-1.10) and the original 0.82 (95%CI: 0.62-1.07)[10]. In this case, the P value from the original analysis (0.144 vs 0.197) is slightly lower.

Target vessel myocardial infarction: The RR of ChatGPT (0.61, 95%CI: 0.42-0.90) was consistent with the original RR (0.61, 95%CI: 0.42-0.89)[10], with P values (0.011 vs 0.011) indicating statistical significance with agreement in both analyses.

Target lesion revascularization: The RR from both analyses were similar (ChatGPT: 0.66, 95%CI: 0.48-0.91; original: 0.67, 95%CI: 0.49-0.91)[10], with ChatGPT’s P-value (0.010), suggesting high concordance with the original P value (0.011).

Target vessel revascularization: Both analyses reported the same RR and CI (0.60, 95%CI: 0.45-0.83), with similar P values (< 0.001).

Stent thrombosis: ChatGPT reported an RR of 0.67 (95%CI: 0.42-1.06) compared to the original 0.63 (95%CI: 0.40-0.99)[10]. The P value from ChatGPT (0.090) indicated non-significance, whereas the original P value (0.046) suggested significance, highlighting a notable difference in the interpretation. Variations in the statistical methods used by ChatGPT and the original analysis could have influenced the P value.

Study 2

MACE: ChatGPT’s HR (0.94, 95%CI: 0.90-0.98) was slightly higher than the original (0.91, 95%CI: 0.87-0.96)[11], with both analyses indicating a significant risk reduction. Although ChatGPT’s P value (0.005) was close to the original (P < 0.0001), the original demonstrated a greater statistical significance.

Cardiovascular death: The HR was similar (ChatGPT: 0.88, 95%CI: 0.81-0.96; original: 0.86, 95%CI: 0.81-0.92)[11], with both analyses showing significant risk reduction. However, the original P value (< 0.0001) indicated a greater level of statistical significance than ChatGPT’s P value (0.0037).

Myocardial infarction: ChatGPT HR (0.97, 95%CI: 0.90-1.05) showed no significant risk reduction, similar to the original (0.95, 95%CI: 0.87-1.04)[11]. However, the ChatGPT P value (0.429) was greater than the original value (0.29), reflecting variations in the statistical interpretation.

Stroke: ChatGPT reported an HR of 0.87 (95%CI: 0.65-1.18) compared to the original 0.99 (95%CI: 0.91-1.07)[11]. However, both P values (ChatGPT: 0.380, original: 0.77) indicated a lack of statistical significance, with slight differences in the HR estimates.

Study 3

LV thrombus formation: The ORs were nearly similar (ChatGPT: 0.37, 95%CI: 0.16-0.86; original: 0.28, 95%CI: 0.11-0.73)[12], with both analyses showing a significant reduction; however, ChatGPT’s P value (0.021) was slightly greater than the original (< 0.01).

Major bleeding events: The OR of 2.82 using ChatGPT (95%CI: 1.30-6.13) was similar to the original 2.85 (95%CI: 1.13-7.24)[12]. Both P values (0.009 ChatGPT vs 0.03 original) indicated statistical significance, with greater significance seen in the analysis by ChatGPT.

Systemic embolism: ChatGPT’s OR (0.42, 95%CI: 0.13-1.29) was almost similar to the original (0.37, 95%CI: 0.12-1.13)[12], with both analyses showing no significant reduction, although ChatGPT’s P value (0.130) was greater than the original (0.08).

Study 4

Periprocedural myocardial infarction (UDMI and UDMI-like definitions) and mortality: The HR of ChatGPT (1.61, 95%CI: 1.32-1.97) was similar to the original HR (1.61, 95%CI: 1.32-1.97)[13], with both analyses demonstrating significant associations. In addition, the ChatGPT P value (< 0.00001) was consistent with the original (< 0.00001).

Periprocedural myocardial infarction (ARC-2 definition) and mortality: Both the ChatGPT and original analyses reported identical HR (2.07, 95%CI: 1.40-3.08)[13] and P values (0.0003), indicating perfect agreement.

Periprocedural myocardial infarction (SCAI and SCAI-like definitions) and mortality: The HR of ChatGPT (3.24, 95%CI: 2.36-4.44) was similar to the original HR (3.24, 95%CI: 2.36-4.44)[13], with both analyses showing significant associations and P values in agreement (< 0.00001).

Study 5

Major vascular complications: The RR calculated by ChatGPT was 2.30 (95%CI: 1.72-3.09), closely matching the original study’s 2.32 (95%CI: 1.73-3.11)[14]. Both analyses demonstrated a significant reduction in cardiac death, with p-values aligned with each other (P < 0.001).

Major bleeding: ChatGPT’s calculated RR of 1.61 (95%CI: 1.27-2.05) was similar to the original 1.61 (95%CI: 1.27-2.05)[14], showing high precision. Both analyses indicated a statistically significant reduction in major bleeding events. Both ChatGPT and the original reported P values of < 0.001.

Aortic annulus rupture: The RR calculated by ChatGPT (4.66, 95%CI: 1.67-13.01) was similar to the original (4.66, 95%CI: 1.67-13.01)[14]. Both analyses reported a significant reduction in all-cause death, with P values of 0.0033 (ChatGPT) and < 0.001 (original), respectively, indicating close agreement.

Cardiac tamponade: ChatGPT reported an RR of 3.0 (95%CI: 1.31-6.89), similar to the original 3.0 (95%CI: 1.31-6.89)[14]. The P value from ChatGPT (0.0095) was similar to the original P value (0.01) after rounding off ChatGPT’s P value.

Minor vascular complications: ChatGPT’s RR of 1.43 (95%CI: 1.00-2.04) was consistent with the original 1.43 (95%CI: 1.00-2.04)[14]. Similarly, the P values (0.052 ChatGPT vs 0.05 original) indicated perfect agreement, with statistical significance.

Stroke: The RR was similar, with ChatGPT reporting 1.41 (95%CI: 0.82-2.40) compared to the original 1.37 (95%CI: 0.81-2.32)[14]. The P value from ChatGPT (0.210) was nearly identical to the original value (0.24), with neither reaching statistical significance.

In-hospital mortality: ChatGPT’s RR of 1.86 (95%CI: 0.74-4.70) was consistent with the original 1.86 (95%CI: 0.74-4.70)[14], and the P values (0.187 vs 0.19) were insignificant and similar in both groups.

30-days mortality: ChatGPT reported an RR of 1.22 (95%CI: 0.55-2.71) compared to the original 1.14 (95%CI: 0.53-2.46)[14]. The P values from ChatGPT (0.628) and the original (0.74) were not significant in either case.

Pacemaker implantation: ChatGPT reported an RR of 0.98 (95%CI: 0.75-1.26) compared to the original 0.91 (95%CI: 0.70-1.18)[14]. The P values from the ChatGPT (0.848) and the original (0.47) were not significant in either analysis.

DISCUSSION

Our study aimed to assess the accuracy and reliability of AI-generated meta-analyses by directly comparing the results produced by the ChatGPT with those from traditional meta-analyses in the field of interventional cardiology. Our comparison across various effect sizes (RR, HR, and OR) across a range of outcome measures revealed a remarkable degree of consistency between the results generated by the ChatGPT and the original studies. Both analyses yielded similar effect sizes and corresponding conclusions, with the exception of the RR for stent thrombosis in the Sreenivasan et al[10] study. In this specific case, the ChatGPT reported a non-significant effect size, whereas the original study concluded that it was statistically significant. This discrepancy highlights the importance of considering both the effect size and level of statistical significance when interpreting the results. It is important to note that although the effect sizes were similar, there were slight differences in the CIs and P values across the analyses. This may be due to chance variations or differences in the analysis methods. More data and larger studies are needed to provide more conclusive evidence.

Interventional cardiology has seen rapid advancements, with numerous studies investigating the efficacy and safety of various interventions such as percutaneous coronary intervention, drug-eluting stents, and novel anticoagulants[10-14]. These studies often produce large volumes of data that must be systematically reviewed and synthesized to guide clinical practice. Traditional meta-analyses are labor-intensive and prone to human error and bias[8]. AI can significantly enhance efficiency by automating time-consuming tasks such as data extraction, synthesis, and statistical analysis, thereby reducing the workload on researchers and accelerating the meta-analytical process.

Rayyan, an AI tool for screening systematic reviews, is frequently used in meta-analytical studies[15]. Robot Reviewer, a machine-learning tool, has been used to assess the risk of bias in clinical studies[16]. The reliability of Robot Reviewers was similar to that of human reviewers and, in some cases, better. ChatGPT has been reported as a powerful tool for intermediate researchers conducting epidemiological studies. However, applying more advanced statistical methods has limitations that require further consideration[17]. A major concern with AI-generated responses is the so-called black box. The black box indicates a system whose internal workings were not transparent. However, the same transparency argument may apply equally to human-generated analysis[18]. Therefore, pinpointing which technical modules of large language models require improvement to enhance the accuracy and reliability of pooled analyses across multiple studies remains challenging. It has yet to be determined whether large language models can effectively perform more complex statistical analyses, such as regression and matched analysis. Enhancing ChatGPT’s statistical coding, data curation, and processing modules would likely refine these capabilities. However, AI-generated responses must still be validated and verified by human expertise. Since AI models rely on high-quality curated datasets, the outcome of one analysis may be biased by the results of prior studies. We propose improving data quality and developing robust training datasets to enhance the reliability of AI-driven analyses. Strengths and Limitations

A major strength of this study was its novelty. To the best of our knowledge, this study is the first to comparatively analyze AI-generated meta-analyses and traditional meta-analyses in the field of cardiology. Additionally, the study compared various effect sizes across diverse outcome measures, providing a thorough assessment of the reliability of the synthesis of the research results.

However, our study had several limitations that must be acknowledged. First, the analysis was confined to the field of cardiology, and the applicability of AI-generated meta-analyses to other medical disciplines remains to be explored. To counteract this and improve generalizability, we include multiple effect measures across a variety of randomly selected journals. Second, although the ChatGPT is a powerful language model, it has inherent limitations in handling complex statistical tasks. The ChatGPT is not designed to independently perform statistical analyses for meta-analyses or other intricate statistical computations. Moreover, it cannot directly access, manipulate, or analyze datasets, nor can it perform statistical computations such as calculating effect sizes or assessing heterogeneity. Third, the inclusion of only five studies at random may introduce potential bias. Finally, ChatGPT lacks the ability to independently validate results, handle missing data, check assumptions, and modify models based on intermediate results. These tasks require specialized statistical software and expert knowledge.

Our study serves as a pilot study and forms the basis for future research. Further exploration of the capabilities of ChatGPT in data synthesis is warranted, based on our results.

CONCLUSION

This study demonstrated the promising potential of ChatGPT in conducting meta-analyses. The overall concordance between the AI-generated and human-conducted analyses suggests that AI can be a valuable tool in this field. However, slight differences in CIs and P values persist, possibly due to different statistical models, or algorithm limitations. We believe that, at present, large language models such as ChatGPT cannot replace the structured and detailed statistical analysis required for rigorous research. Further research and development are essential for integrating AI into mainstream medical research and practice.