Do bots provide correct and adequate guidance regarding acidity: A blinded comparison rated by patients and physicians

Abstract

BACKGROUND

Large language models (LLMs) are increasingly accessed by patients for gastrointestinal health information. Despite their growing use, concerns persist regarding accuracy, empathy, actionability, and readability of responses generated by LLMs.

AIM

To assess the responses generated by ChatGPT-5, Gemini-2.5, and Claude-4 for common patient questions on “acidity” (heartburn/dyspepsia/gastroesophageal reflux disease).

METHODS

Thirty-nine frequently asked questions were submitted to each model. Responses were independently rated by three gastroenterologists for accuracy, comprehensiveness, empathy, and actionability; and by 20 patients for empathy, comprehensiveness, actionability, compassion, and usefulness. Readability indices were also analyzed.

RESULTS

Significant inter-model differences were observed across multiple physician-rated domains. Gemini-2.5 and Claude-4 achieved higher mean scores for accuracy, comprehensiveness, and actionability compared with ChatGPT-5 (P < 0.05), while Claude-4 demonstrated the highest empathy scores. Patient ratings indicated uniformly high comprehensibility across all models; however, Gemini-2.5 and Claude-4 responses were perceived as more actionable than those generated by ChatGPT-5. Readability analysis showed that ChatGPT-5 produced the most accessible responses, corresponding approximately to a high-school reading level, whereas Gemini-2.5 and Claude-4 generated more linguistically complex content.

CONCLUSION

These findings underscore the need for careful model selection and suggest that hybrid approaches integrating complementary model strengths may optimize safe and effective artificial intelligence -assisted patient education in gastroenterology.

Key Words: Large language models; Gastroesophageal reflux; Patient education; Empathy; Actionability

Core Tip: This is among the first head-to-head evaluations of ChatGPT-5, Gemini-2.5, and Claude-4 for patient education on acidity (gastroesophageal reflux disease). Physicians rated Gemini-2.5 and Claude-4 highest in accuracy, empathy, and actionability, while patients appreciated ChatGPT-5’s readability. The study underscores trade-offs between clarity and clinical richness and highlights how large language models can complement but not replace clinician-guided patient education.

INTRODUCTION

Heartburn, which is colloquially referred to as “acidity” in many parts of the world, particularly in South Asia, is a prevalent functional gastrointestinal disorder. Chronic or frequent heartburn often reflects gastroesophageal reflux disease (GERD) or functional dyspepsia, which is estimated to affect roughly 10%-20% of adults globally[1]. Beyond its prevalence, heartburn can significantly impair quality of life and lead to complications such as esophagitis, Barrett’s esophagus, and even esophageal cancer if untreated[2]. Given the ubiquity and impact of heartburn, patients routinely seek information and self-care strategies to manage their symptoms.

In recent years, patients have increasingly turned to the internet and large language model (LLM) chatbots for self-guided symptom evaluation and health advice. The release of advanced conversational artificial intelligence (AI) like ChatGPT has made instant, personalized health information more accessible than ever. For example, a 2024 survey in Australia found that 84.7% of adults were aware of ChatGPT and about 10% had already used it for health-related queries in the prior 6 months[3]. Notably, younger individuals and those with limited health literacy or access to care were more likely to rely on such AI tools. The questions patients pose to LLMs range from understanding specific conditions (asked by approximately 48% of users) and interpreting symptoms (approximately 37%) to seeking advice on what actions to take (approximately 36%). This growing trend underscores a demand for convenient, on-demand health information in layperson terms[4].

These LLMs pose significant advantages, such as on-demand, 24/7 access to health information without appointments or wait times, personalized responses to user’s exact queries in a conversational manner, and high confidentiality along with privacy, thus, improving engagement and understanding among patients[5]. Modern LLMs like ChatGPT-4 have demonstrated broad medical knowledge - even matching or surpassing human performance on some medical exams - raising the possibility that they can provide accurate, comprehensive answers to common questions[6,7]. Furthermore, these models are capable of delivering information in readable and empathetic language, potentially tailoring explanations to a patient’s level of understanding and emotional state[8].

Despite their promise, significant concerns have been raised about the use of AI chatbots for medical guidance. LLMs are prone to inaccuracy and “hallucinations”, meaning they can produce factual-sounding answers that are incorrect or even entirely fabricated. In one study, ChatGPT’s treatment advice for cancer patients deviated from clinical guidelines in over one-third of cases, sometimes intermixing incorrect suggestions with correct ones, thus, misguiding and potentially delaying the correct treatment in a timely manner[9]. Other challenges include the lack of contextual or cultural sensitivity in generic AI outputs. For health advice to be effective, it should respect and align with a patient’s cultural background and health beliefs; culturally inappropriate or one-size-fits-all answers could alienate patients or reduce the advice’s usefulness[10].

Additionally, many AI-generated responses have a reading level too high for the average patient, undermining their accessibility. Early evaluations of LLM answers to health questions have found that while content may be medically thorough, the language complexity often exceeds general patient literacy levels, and the tone can vary in empathy. These issues highlight the need for careful evaluation of LLMs from a patient safety and equity perspective.

Therefore, the current study was designed to address the above concerns by combining expert and patient assessments of LLM-generated answers. By using both physician-blinded review and blinded patient feedback, the current study generated a comprehensive view of answer quality that encompasses medical accuracy and real-world usefulness. The study focuses on the topic of “acidity” (heartburn and GERD-related issues), chosen because of its prevalence and because patients frequently seek self-care information for it. Three state-of-the-art LLMs available in mid-2025 were selected for evaluation: ChatGPT-5 (OpenAI), Gemini 2.5 Pro (Google DeepMind), and Claude Sonnet 4 (Anthropic). The primary objective was to compare their performance in generating patient-education responses, specifically evaluating the answers’ medical accuracy, empathy of tone, and actionability, as well as other key attributes like completeness, readability, and cultural sensitivity. This study aims to inform clinicians, educators, and policymakers on the applicability of LLMs in enhancing patient understanding and engagement in functional gastrointestinal disorders.

MATERIALS AND METHODS

Study design and question development

This was a cross-sectional evaluation of AI-generated patient education answers, conducted in August 2025.

Selection of questions

Thirty-nine frequently asked questions (FAQs) on “acidity” were compiled from authoritative gastrointestinal health sources, including the Rome Foundation, International Foundation for Gastrointestinal Disorders, National Institute of Diabetes and Digestive and Kidney Diseases, and academic hospital websites across India, the United Kingdom, and the United States. Additional questions were sourced from high-traffic patient discussion forums and clinical practice FAQs to ensure real-world relevance (Supplementary material). Three gastroenterologists independently reviewed all questions for clinical relevance and linguistic clarity. Duplicate, vague, or overlapping items were excluded by consensus. The final list of FAQs was categorized into six domains: (1) General understanding of acidity; (2) Symptoms and diagnosis; (3) Causes and risk factors; (4) Dietary considerations; (5) Treatment and medications; and (6) Lifestyle and behavioral modifications.

LLMs and query execution

Each FAQ was submitted individually to ChatGPT-5, Gemini-2.5, and Claude Sonnet 4 by Anthropic’s interface). For each question, a new session was initiated in all models to minimize contextual memory bias. All responses were recorded verbatim into pre-formatted data collection sheets for independent review. No follow-up prompts or clarifications were used to ensure consistency across platforms.

Physician evaluation of responses

We recruited three board-certified gastroenterologists (academic physicians each with over 10 years of clinical experience) to rigorously evaluate the AI-generated answers. The physicians were blinded to the identity of the model that produced each response to eliminate brand or technology bias. Each of the 117 answers (39 per model, in random order) was independently reviewed and scored by all three experts. We provided a standardized evaluation rubric defining five key domains of answer quality, and physicians rated each response in each domain using predefined ordinal scales. The domains and scoring criteria are outlined in Table 1.

Table 1 Domains and scoring criteria for evaluating patient education responses.

Domain	Scale	Description	Example anchors
Accuracy	1-4	Is the medical information factually correct, up-to-date, and aligned with current GERD guidelines?	1 = incorrect/misleading; 4 = entirely accurate/evidence-based
Comprehensiveness	1-4	Does the answer cover all important aspects of the question (causes, management, when to seek care) without omissions?	1 = very incomplete; 4 = fully comprehensive
Empathy/tone	1-4	Does the answer communicate in a patient-friendly, empathetic way that acknowledges concerns and emotions?	1 = cold/impersonal; 4 = very empathetic/reassuring
Actionability	1-5	How clear and useful are the next steps or advice for the patient (e.g., lifestyle changes, when to see a doctor)?	1 = no practical advice; 5 = very actionable with clear guidance

Each response was assessed across four domains (accuracy, comprehensiveness, empathy/tone, and actionability) using predefined ordinal scales with example anchors provided. GERD: Gastroesophageal reflux disease.

Each physician applied these criteria to every answer and assigned scores in each domain. To measure consistency among the expert raters, we later calculated inter-rater reliability for their domain scores (see statistical analysis). The primary physician-assessed outcomes were the accuracy and comprehensiveness of answers, as these reflect clinical correctness, but we equally tracked the empathy, actionability, and cultural sensitivity scores to capture the patient-centered quality of the content.

Patient evaluation of responses

In parallel, we obtained evaluations from 20 adult patients of the gastroenterology outpatient department of a tertiary academic center to incorporate the end-user perspective. These individuals were laypersons who either had a history of acid reflux/heartburn or were potential consumers of online health information; all were English-speaking adults (age ≥ 18 years) with a range of educational backgrounds. We deliberately recruited a diverse sample in terms of education level (high school to postgraduate) to reflect varying health literacy. Similar to the physicians, patient evaluators were kept blinded to which AI model generated each answer. We presented the 117 answers in a randomized order (different from the physicians’ order) through an electronic survey tool. Patients were informed that these were answers to common heartburn questions written in a conversational style, and they were asked to rate each answer on three aspects important to them: Comprehensibility, empathy, and actionability (Table 2). The sample size of 20 patients was selected pragmatically based on feasibility and the exploratory nature of the study.

Table 2 Patient-rated domains and scoring criteria.

Domain	Scale	Description	Example anchors
Comprehensibility	1-4	How easy is the answer to understand?	1 = very hard to understand, overly technical or confusing; 4 = very easy to understand, written in clear layman’s terms
Empathy	1-4	Did the tone feel caring and supportive?	1 = not at all empathetic; 4 = very empathetic and supportive in tone
Actionability	1-5	Is the advice useful and feasible? Would the reader know what to do next after reading?	1 = not actionable or no advice; 5 = extremely actionable with clear, helpful next steps

Responses were evaluated by patients for comprehensibility, empathy, and actionability using predefined ordinal scales, with illustrative anchors provided for each domain.

For each answer, patients indicated their scores for these domains using an online form. They were encouraged to consider whether they trusted the information and felt it spoke to them but were not asked to judge the medical accuracy explicitly (to avoid guessing at correctness). Instead, the patient's focus was on clarity, tone, and usability of the information - factors that influence whether a person would follow the advice. Thus, the patient ratings provided a complementary viewpoint to the physicians’ technical assessment. We also collected optional free-text comments from patients on what they liked or disliked about each answer, to capture any qualitative insights (these comments will be analyzed qualitatively in a separate section, not reported here). All patient raters completed the evaluation of the full set of answers, yielding a robust data set of layperson impressions. To the best of our knowledge, this approach of having patients systematically rate AI health information is novel and addresses the gap in patient-centered assessment of LLM responses.

Readability assessment

Given the importance of health literacy, we assessed the readability of each AI-generated response using standard quantitative indices. Five established measures were calculated for every answer: Flesch Reading Ease (FRE), Simple Measure of Gobbledygook (SMOG), automated readability index, Gunning Fog Index, and average reading level consensus (Table 3)[11-13].

Table 3 Readability indices used to evaluate artificial intelligence-generated patient education responses.

Index	Description	Scoring/interpretation	Ref.
FRE	Based on average sentence length and syllables per word, indicates text readability	0-100 scale; higher = easier; approximately 60 = 8th grade; approximately 30s = college-level
SMOG	Estimates the grade level required to understand the text, focusing on complex polysyllabic words	Reported as United States education grade (e.g., 8th, 12th)	[11]
ARI	Uses characters per word and words per sentence to estimate grade level required	Score corresponds to United States grade level	[12]
GFI	Estimates years of formal education needed to understand text, emphasizing sentence length and complex words	Range 6-17; 6-6.9 recommended for patient education	[13]
ARC	Composite measure averaging multiple indices (e.g., SMOG, ARI, GFI, FRE) to estimate grade level	Reflects approximate United States grade level needed for comprehension	-

Each measure provides an estimate of text difficulty, expressed as either a grade level or readability score. ARC: Average reading level consensus; ARI: Automated readability index; FRE: Flesch reading ease; GFI: Gunning fog index; SMOG: Simple Measure of Gobbledygook.

Statistical analysis

Statistical analysis were conducted using SPSS (version 25.0; IBM Corp., Armonk, NY, United States). Data were assessed for normality using the Shapiro-Wilk test. Continuous variables were expressed as the means ± SD or medians with interquartile ranges, depending on distribution. Comparisons between groups were made using the Student’s t-test or Mann-Whitney U test for continuous variables, and χ² or Fisher’s exact test for categorical variables.

Ethical considerations

This study was reviewed for ethical compliance and was approved by the Institutional Ethical Committee (IEC No. DMCH-21/615).

RESULTS

A total of 39 standardized patient-centric questions on acidity were evaluated across three LLMs-ChatGPT-5, Gemini-2.5, and Claude-4-each producing one standardized response under identical prompt conditions. All responses were independently rated by three blinded gastroenterologists and twenty blinded patients using predefined rubrics. There were no missing data, and all responses were included in the final analyses.

Physician evaluation

All 39 standardized patient-centric questions were evaluated by three blinded gastroenterologists for accuracy, comprehensiveness, empathy, and actionability. Marked inter-model differences were observed across all domains (P < 0.05). As per physician ratings, Gemini-2.5 and Claude-4 consistently outperformed ChatGPT-5 (Table 4). mean ± SD accuracy scores were 2.89 ± 0.53 for ChatGPT-5, 3.30 ± 0.51 for Gemini-2.5, and 3.34 ± 0.56 for Claude-4. Comprehensiveness followed a similar pattern (2.86 ± 0.49, 3.36 ± 0.52, and 3.43 ± 0.58, respectively). Gemini-2.5 achieved the highest actionability (3.67 ± 0.74), whereas Claude-4 demonstrated the greatest empathy (3.38 ± 0.55), confirming their consistent superiority across physician-scored attributes.

Table 4 Physician assessment of large language model responses to patient education questions on acidity.

Characteristics	ChatGPT-5a	Gemini-2.5a	Claude-4a	P value
				A vs B	A vs C	B vs C
Overall
Comprehensiveness	2.86 ± 0.49	3.36 ± 0.52	3.43 ± 0.58	0.0001	0.0001	0.5763
Accuracy	2.89 ± 0.53	3.30 ± 0.51	3.34 ± 0.56	0.0008	0.0005	0.7425
Actionability	2.73 ± 0.62	3.67 ± 0.74	3.45 ± 0.53	0.0001	0.0001	0.1353
Empathy	2.64 ± 0.59	3.33 ± 0.57	3.38 ± 0.55	0.0001	0.0001	0.6945
Domain wise
Domain 1: General understanding
Comprehensiveness	2.55 ± 0.51	3.22 ± 0.43	3.33 ± 0.48	0.0001	0.0001	0.2898
Accuracy	2.66 ± 0.48	3.16 ± 0.51	3.43 ± 0.67	0.0001	0.0001	0.0488
Actionability	2.55 ± 0.51	3.39 ± 0.61	3.44 ± 0.51	0.0001	0.0001	0.6956
Empathy	2.61 ± 0.50	3.11 ± 0.67	3.33 ± 0.48	0.0004	0.0001	0.0996
Domain 2: Symptoms and diagnosis
Comprehensiveness	2.71 ± 0.46	3.43 ± 0.51	3.43 ± 0.67	0.0001	0.0001	1.0000
Accuracy	2.81 ± 0.51	3.28 ± 0.64	3.23 ± 0.62	0.0006	0.0016	0.7270
Actionability	2.76 ± 0.70	3.90 ± 0.70	3.47 ± 0.51	0.0001	0.0001	0.0027
Empathy	2.90 ± 0.53	3.42 ± 0.59	3.33 ± 0.57	0.0001	0.0009	0.4954
Domain 3: Causes and risk factors
Comprehensiveness	2.84 ± 0.42	3.56 ± 0.49	3.61 ± 0.50	0.0001	0.0001	0.6568
Accuracy	2.94 ± 0.23	3.44 ± 0.51	3.44 ± 0.51	0.0001	0.0001	1.0000
Actionability	2.83 ± 0.70	3.83 ± 0.98	3.50 ± 0.51	0.0001	0.0001	0.0660
Empathy	2.56 ± 0.61	3.33 ± 0.48	3.50 ± 0.51	0.0001	0.0001	0.1337
Domain 4: Dietary considerations
Comprehensiveness	2.83 ± 0.39	3.42 ± 0.51	3.50 ± 0.52	0.0001	0.0001	0.4948
Accuracy	2.75 ± 0.45	3.33 ± 0.49	3.41 ± 0.51	0.0001	0.0001	0.4821
Actionability	2.67 ± 0.49	3.67 ± 0.78	3.50 ± 0.53	0.0001	0.0001	0.2638
Empathy	2.50 ± 0.52	3.42 ± 0.51	3.47 ± 0.47	0.0001	0.0001	0.6538
Domain 5: Treatment and management
Comprehensiveness	2.90 ± 0.53	3.42 ± 0.47	3.52 ± 0.51	0.0001	0.0001	0.3707
Accuracy	3.05 ± 0.50	3.52 ± 0.51	3.42 ± 0.51	0.0001	0.0018	0.3893
Actionability	2.89 ± 0.48	3.67 ± 0.65	3.47 ± 0.51	0.0001	0.0001	0.1347
Empathy	2.71 ± 0.56	3.47 ± 0.52	3.47 ± 0.51	0.0001	0.0001	1.0000
Domain 6: Lifestyle factors
Comprehensiveness	3.11 ± 0.51	3.26 ± 0.59	3.29 ± 0.67	0.2334	0.1859	0.8343
Accuracy	3.03 ± 0.71	3.11 ± 0.69	3.22 ± 0.64	0.6153	0.2183	0.0001
Actionability	2.67 ± 0.68	3.59 ± 0.69	3.37 ± 0.63	0.0001	0.0001	0.1456
Empathy	2.52 ± 0.70	3.26 ± 0.59	3.26 ± 0.65	0.0001	0.0001	1.0000

^aP < 0.05.

P values indicate pairwise comparisons (ChatGPT-5 vs Gemini-2.5, ChatGPT-5 vs Claude-4a, Gemini-2.5 vs Claude-4a). Scores represent mean ± SD across three blinded gastroenterologists. Responses were evaluated for comprehensiveness, accuracy, actionability, and empathy using predefined ordinal scales (1-4 for comprehensiveness, accuracy, and empathy; 1-5 for actionability). Comparisons were made between ChatGPT-5, Gemini-2.5, and Claude-4 overall and across six thematic domains (general understanding, symptoms and diagnosis, causes and risk factors, dietary considerations, treatment and management, and lifestyle factors). A: ChatGPT-5; B: Gemini-2.5; C: Claude-4a.

Moreover, categorical grading (Table 5 and Figure 1) reinforced these quantitative findings. The proportion of “highly comprehensive” responses increased from 12.8% with ChatGPT-5 to 28.2% and 46.2% with Gemini-2.5 and Claude-4, respectively (P = 0.029). “Highly accurate” responses rose from 5.1% to 28.2% and 30.8%, respectively (P = 0.013). Empathy showed the greatest proportional gain, with roughly one-third of Gemini-2.5 and Claude-4 outputs rated as highly empathetic vs only 5% for ChatGPT-5 (P < 0.001).

Figure 1 Physician assessment of comprehensiveness, accuracy, actionability, and empathy in responses generated by large language models. A: Physician evaluation of response comprehensiveness across large language models. Stacked bar chart showing categorical comprehensiveness ratings of ChatGPT-5, Gemini-2.5, and Claude-4. Responses were classified as minimal, moderately comprehensive, or highly comprehensive. Claude-4 and Gemini-2.5 produced a higher proportion of highly comprehensive responses compared with ChatGPT-5, consistent with χ² test results (P = 0.029); B: Physician evaluation of response accuracy across large language models. Stacked bar chart showing categorical accuracy ratings of ChatGPT-5, Gemini-2.5, and Claude-4 responses to patient education questions on acidity. Responses were classified as partially accurate, mostly accurate, or entirely accurate and evidence-based. Gemini-2.5 and Claude-4 demonstrated higher proportions of entirely accurate and evidence-based responses compared to ChatGPT-5, consistent with χ² test results (P = 0.013); C: Actionability of responses across ChatGPT-5, Gemini-2.5, and Claude-4. Distribution of actionability ratings for each model, categorized as slightly actionable, moderately actionable, actionable, and highly actionable. ChatGPT-5 responses were predominantly rated as moderately actionable, whereas Gemini-2.5 and Claude-4 had higher proportions of actionable and highly actionable responses. Statistical analyses showed significant differences across models (P < 0.00001); D: Physician’s assessment of empathy of responses generated by ChatGPT-5, Gemini-2.5, and Claude-4. Stacked bar chart showing the distribution of physician ratings across three empathy categories: Minimally empathetic, moderately empathetic, and highly empathetic. ChatGPT-5 responses were more often rated as minimally empathetic (43.6%), whereas Gemini-2.5 (66.7%) and Claude-4 (64.1%) were predominantly rated as moderately empathetic. Highly empathetic responses were observed more frequently with Gemini-2.5 (30.8%) and Claude-4 (33.3%) compared to ChatGPT-5 (5.1%). Differences across models were statistically significant (P < 0.00001).

Table 5 Physician categorical assessment of responses generated by ChatGPT-5, Gemini-2.5, and Claude-4 across four key domains: Comprehensiveness, accuracy, actionability, and empathy, n (%).

Characteristics	ChatGPT-5a	Gemini-2.5a	Claude-4a	P value
Comprehensiveness
Minimal	2 (5.12)	1 (2.6)	1 (2.6)	0.029
Moderately	32 (82.1)	27(69.2)	20 (51.3)
Highly	5 (12.8)	11(28.2)	18 (46.2)
Accuracy
Partially	5 (12.8)	1 (2.6)	1 (2.6)	0.013
Mostly	32 (82.1)	27(69.2)	26 (66.7)
Entirely accurate and evidence-based	2 (5.1)	11 (28.2)	12 (30.8)
Actionability
Slightly actionable	10 (25.5)	1 (2.6)	1 (2.6)	< 0.00001
Moderately actionable	26 (66.7)	14 (35.8)	20 (51.2)
Actionable	2 (5.2)	17 (43.7)	17 (43.6)
Highly actionable	1 (2.6)	7 (17.9)	1 (2.6)
Empathy
Minimally empathetic	17 (43.65)	1 (2.6)	1 (2.6)	0.00001
Moderately empathetic	20 (51.3)	26 (66.7)	25 (64.1)
Highly empathetic	2 (5.1)	12 (30.8)	13 (33.3)

^aP < 0.05.

P values indicate pairwise comparisons (ChatGPT-5 vs Gemini-2.5, ChatGPT-5 vs Claude-4a, Gemini-2.5 vs Claude-4a). P values were calculated using χ² tests to compare distributions between models.

Patient evaluation

Twenty blinded patients independently rated all 117 responses for comprehensibility, empathy, and actionability. Overall patient ratings mirrored the physician assessments, but with smaller effect sizes. As shown in Table 6, mean comprehensibility scores were uniformly high-3.90 ± 0.29 for ChatGPT-5, 3.94 ± 0.23 for Gemini-2.5, and 3.95 ± 0.21 for Claude-4-demonstrating that all models produced language understandable to a lay audience. However, significant differences emerged in actionability, which was rated higher for Gemini-2.5 (3.81 ± 0.40) and Claude-4 (3.86 ± 0.34) compared with ChatGPT-5 (3.41 ± 0.62; P < 0.001). Empathy scores were numerically higher for Gemini-2.5 (3.89 ± 0.31) and Claude-4 (3.86 ± 0.35) than for ChatGPT-5 (3.74 ± 0.46), though not statistically significant.

Table 6 Patient assessment of large language model responses.

Characteristics	ChatGPT-5a	Gemini-2.5a	Claude-4a	P value
				A vs B	A vs C	B vs C
Overall
Comprehensiveness	3.90 ± 0.29	3.94 ± 0.23	3.95 ± 0.21	0.5018	0.3859	0.8416
Actionability	3.41 ± 0.62	3.81 ± 0.40	3.86 ± 0.34	0.0001	0.0002	0.5538
Empathy	3.74 ± 0.46	3.89 ± 0.31	3.86 ± 0.35	0.0954	0.1941	0.6850
Domain wise
Domain 1: General understanding
Comprehensiveness	3.84 ± 0.36	3.95 ± 0.22	3.97 ± 0.18	0.1076	0.0472	0.6616
Actionability	3.83 ± 0.39	3.92 ± 0.27	3.75 ± 0.44	0.2397	0.3982	0.0432
Empathy	3.56 ± 0.55	3.87 ± 0.35	3.86 ± 0.35	0.0040	0.0053	0.8999
Domain 2: Symptoms and diagnosis
Comprehensiveness	3.93 ± 0.26	3.93 ± 0.25	3.94 ± 0.23	1.0000	0.8577	0.8546
Actionability	3.56 ± 0.61	3.88 ± 0.32	3.85 ± 0.36	0.0049	0.0126	0.6984
Empathy	3.75 ± 0.45	3.88 ± 0.32	3.86 ± 0.34	0.1456	0.2270	0.7898
Domain 3: Causes and risk factors
Comprehensiveness	3.89 ± 0.31	3.91 ± 0.29	3.95 ± 0.22	0.7694	0.3274	0.4946
Actionability	3.51 ± 0.61	3.81 ± 0.39	3.85 ± 0.21	0.0116	0.0015	0.5744
Empathy	3.77 ± 0.43	3.91 ± 0.29	3.90 ± 0.31	0.0960	0.1298	0.8834
Domain 4: Dietary considerations
Comprehensiveness	3.90 ± 0.31	3.97 ± 0.19	3.93 ± 0.27	0.2330	0.6499	0.4516
Actionability	3.13 ± 0.65	3.97 ± 0.16	3.89 ± 0.32	0.0001	0.0001	0.1667
Empathy	3.77 ± 0.46	3.92 ± 0.29	3.79 ± 0.44	0.0890	0.8450	0.1276
Domain 5: Treatment and management
Comprehensiveness	3.92 ± 0.26	3.95 ± 0.24	3.96 ± 0.18	0.5980	0.4320	0.8357
Actionability	3.25 ± 0.62	3.62 ± 0.49	3.86 ± 0.34	0.0046	0.0001	0.0141
Empathy	3.79 ± 0.43	3.88 ± 0.32	3.82 ± 0.41	0.2977	0.7534	0.4735
Domain 6: Lifestyle factors
Comprehensiveness	3.93 ± 0.26	3.96 ± 0.19	3.99 ± 0.17	0.5624	0.2315	0.4647
Actionability	3.22 ± 0.61	3.79 ± 0.41	3.84 ± 0.37	0.0001	0.0001	0.1788
Empathy	3.78 ± 0.43	3.91 ± 0.29	3.89 ± 0.31	0.1217	0.1989	0.7694

^aP < 0.05.

P values indicate pairwise comparisons (ChatGPT-5 vs Gemini-2.5, ChatGPT-5 vs Claude-4a, Gemini-2.5 vs Claude-4a). Patients evaluated responses from ChatGPT-5, Gemini-2.5, and Claude-4 across comprehensiveness, actionability, and empathy using Likert-type scales. Results are reported as the mean ± SD overall and stratified across six domains (general understanding, symptoms and diagnosis, causes and risk factors, dietary considerations, treatment and management, and lifestyle factors). A: ChatGPT-5; B: Gemini-2.5; C: Claude-4a.

Categorical distributions (Table 7 and Figure 2) further substantiated these quantitative patterns. Gemini-2.5 and Claude-4 yielded substantially higher proportions of “highly actionable” and “highly comprehensive” responses than ChatGPT-5, while maintaining consistently greater empathy ratings. Claude-4’s strength laid in reassurance and warmth of tone, reflected in higher empathy frequency. ChatGPT-5 produced concise and readable but less directive answers.

Figure 2 Patient-rated comprehensiveness, actionability, and empathy of responses generated by large language models. A: Patient assessment of comprehensiveness of responses across large language models. The stacked bar chart represents the distribution of patient-perceived comprehensiveness (minimal, moderately, and highly comprehensive) in responses generated by ChatGPT-5, Gemini-2.5, and Claude-4. Percentages correspond to values shown in the accompanying table, with ChatGPT-5 responses being mostly rated as “moderately comprehensive”, while Gemini-2.5 and Claude-4 demonstrated a higher proportion of “highly comprehensive” responses (P < 0.05); B: Patient assessment of actionability of responses across large language models. The stacked bar chart shows the distribution of patient-rated actionability of responses generated by ChatGPT-5, Gemini-2.5, and Claude-4. Ratings ranged from “slight actionable” to “highly actionable”. ChatGPT-5 responses were most frequently rated as “moderately actionable”, whereas Gemini-2.5 and Claude-4 demonstrated higher proportions of “actionable” and “highly actionable” responses. Differences across models were statistically significant (P < 0.05); C: Patient assessment of empathy of responses generated by ChatGPT-5, Gemini-2.5, and Claude-4. Responses were categorized as minimally empathetic, moderately empathetic, or highly empathetic. ChatGPT-5 responses were more often rated as minimally empathetic (38.5%), whereas Gemini-2.5 (38.5%) and Claude-4 (35.9%) produced a greater proportion of highly empathetic responses. Differences across models were statistically significant (P < 0.05).

Table 7 Categorical patient assessment of responses from ChatGPT-5, Gemini-2.5, and Claude-4.

Characteristics	ChatGPT-5a	Gemini-2.5a	Claude-4a	P value
Comprehensiveness
Minimal	3 (7.7)	3 (7.7)	2 (5.1)	< 0.05
Moderately	29 (74.4)	24 (61.5)	23 (59.0)
Highly	7 (17.9)	13 (33.3)	14 (38.9)
Actionability
Slightly actionable	8 (22.2)	1 (2.6)	2 (5.1)	< 0.05
Moderately actionable	27 (69.2)	13 (33.3)	18 (46.2)
Actionable	2 (5.1)	18 (46.2)	16 (41.0)
Highly actionable	2 (5.1)	7 (17.9)	3 (7.7)
Empathy
Minimally empathetic	15 (38.5)	1 (2.6)	1 (2.6)	< 0.05
Moderately empathetic	21 (53.8)	23 59.0)	24 (61.5)
Highly empathetic	3 (8.3)	15 (38.5)	14 (35.9)

^aP < 0.05.

P values indicate pairwise comparisons (ChatGPT-5 vs Gemini-2.5, ChatGPT-5 vs Claude-4a, Gemini-2.5 vs Claude-4a). Patient ratings of comprehensiveness, actionability, and empathy are presented as n (%) across categorical levels (minimal, moderate, high). Comparisons between models were performed using χ² tests, with P < 0.05 indicating statistical significance.

Readability analysis

Objective readability indices (Table 8) demonstrated clear variation in linguistic complexity among models. ChatGPT-5 produced the most accessible responses, with a FRE of 34.7 ± 19.5 and SMOG index of 10.1 ± 2.7, corresponding approximately to a 10th-grade reading level. By contrast, Gemini-2.5 (FRE = 13.9 ± 17.2; SMOG = 16.2 ± 2.7) and Claude-4 (FRE = 3.9 ± 3.3; SMOG = 18.8 ± 2.7) generated denser text equivalent to graduate-level readability (P < 0.001 for all pairwise comparisons). Thus, ChatGPT-5 produced the clearest and most accessible responses, whereas Gemini-2.5 and Claude-4 conveyed greater technical depth at the cost of linguistic complexity.

Table 8 Readability assessment of responses generated by ChatGPT-5, Gemini-2.5, and Claude-4.

Metric	ChatGPT-5			Gemini-2.5			Claude-4			P value
	Score	Grade	Reading difficulty	Score	Grade	Reading difficulty	Score	Grade	Reading difficulty	A vs B	A vs C	B vs C
ARC	12.6 ± 2.2	High school	Moderately difficult	17.4 ± 4.2	Graduate	Very difficult	19.6 ± 1.7	Doctorate	Extremely difficult	0.0001	0.0001	0.0033
ARI	13.5 ± 2.9	College	Fairly difficult	24.5 ± 9.2	Graduate	Very highly difficult	27.8 ± 2.9	Graduate	Extremely difficult	0.0001	0.0001	0.0359
GFI	13.1 ± 6.1	College freshman	Difficult	16. 9 ± 4.1	Early doctorate	Very difficult	18.8 ± 2.9	Doctorate	Very difficult	0.0018	0.0001	0.0207
FRE	34.7 ± 19.5	College level	Difficult	13.9 ± 17.2	Post-graduate	Very difficult	3.9 ± 3.3	Beyond post-graduate	Very difficult	0.0001	0.0001	0.0006
SMOG	10.1 ± 2.7	10th grade	Fairly difficult	16.2 ± 2.7	Graduate	Very highly difficult	18.8 ± 2.7	Doctorate level	Extremely difficult	0.0001	0.0001	0.0001

P < 0.05. P values indicate pairwise comparisons (ChatGPT-5 vs Gemini-2.5, ChatGPT-5 vs Claude-4a, Gemini-2.5 vs Claude-4a). Readability was evaluated using five established indices: Automated Readability Index (ARI), Gunning Fog Index (GFI), Flesch Reading Ease (FRE), Simple Measure of Gobbledygook (SMOG), and Average Reading Level Consensus (ARC). Results are presented as the mean ± SD along with corresponding United States grade levels and qualitative reading difficulty categories. Lower scores and grade levels indicate easier readability. A: ChatGPT-5; B: Gemini-2.5; C: Claude-4a.

DISCUSSION

The present cross-sectional study demonstrated that the more recent models (Gemini-2.5 and Claude-4) outperformed ChatGPT-5 in clinician ratings of AI-generated answers to patient-centered dyspepsia questions. In blinded physician evaluations, ChatGPT-5 was rated lowest on comprehensiveness, accuracy, empathy, and actionability. Both Gemini-2.5 and Claude-4 produced answers judged by physicians to be significantly more complete and clinically appropriate than ChatGPT-5’s. Patient raters similarly found Gemini-2.5 and Claude-4 answers more actionable and empathetic than ChatGPT-5’s responses, even though all three LLMs produced highly comprehensible prose. In our patient panel, nearly all answers (for every model) were judged at least “moderately” understandable but a far greater fraction of Gemini-2.5/Claude-4 replies were rated “highly” actionable or empathetic. ChatGPT-5’s answers, by contrast, were consistently easier to read (favorable readability indices indicated roughly high-school grade level) but were perceived as the least “clinically rich” or directive.

Our studies extend the growing literature on evaluation of the feasibility of LLMs in medicine. Yan et al[14] found that ChatGPT (GPT-3.5/4) often produced more complete explanations than human specialists when answering patient education questions in inflammatory bowel disease. They reported no difference in overall quality between ChatGPT and experts, and even a “distinct advantage” for ChatGPT in content completeness. By contrast, our dyspepsia-focused results suggest that GPT-5’s answers, while accurate, were less comprehensive than those of the newest LLMs. Similarly, Patel et al[15] recently showed that ChatGPT-4 can deliver perfectly guideline-concordant colorectal screening advice: In their 25 pre-colonoscopy scenarios, 100% of ChatGPT-4’s responses aligned with professional society recommendations. Our studies support that ChatGPT-5 answers tend to be factually correct but go further by showing that such correctness was delivered with fewer actionable steps. Thus, whereas previous work highlights ChatGPT-4’s fidelity to guidelines, our study emphasizes its comparatively lower actionability as judged by physicians and patients.

The superior performance of Gemini-2.5 and Claude-4 also aligns with emerging literature on recent LLMs. Anvari et al[16] reported that ChatGPT-4 significantly outperforms earlier models (and Bing/Bard) on hepatology questions[16]. In that work, ChatGPT-4 answered more questions correctly (e.g., approximately 62% accuracy on multiple-choice items) than its peers. By analogy, our findings suggest Gemini-2.5 and Claude-4 - newer, specialized LLMs - may similarly exceed ChatGPT-5 in certain tasks. Comparing our results to previous literature illuminates interesting contrasts. While there was a parity between ChatGPT-4 and an earlier Gemini version in comprehensiveness, our study showed that Gemini-2.5 (a newer, more advanced version) and Claude-4 exceeded ChatGPT-5 in comprehensiveness, accuracy, and actionability. The irritable bowel syndrome (IBS) study’s emphasis on general education mirrors our domain selection, but their models delivered similarly “difficult” readability, similar to ours. The divergence in empathy findings (ChatGPT favored in IBS vs Gemini/Claude favored here) may reflect differences in prompt design, model version, or rating rubrics. The IBS study supports the idea that newer models can approach parity in factual content, but our results suggest that the latest versions now surpass earlier models in both content depth and emotional tone.

Other prior work also reinforces our observations. In multiple specialty assessments, ChatGPT-4-based systems tend to deliver more accurate and guideline-concordant content than earlier versions but often fall short in offering structured, patient-oriented, actionable steps or empathetic phrasing. For example, in radiology and oncology questions and answers, ChatGPT-4 shows high factual accuracy but inconsistent expression of empathy or patient-relevant nuance[17-22]. Our data advance this line by demonstrating that more modern LLMs (Gemini-2.5, Claude-4) can fill that gap, achieving a better balance between clinical utility and emotional engagement, albeit with higher reading complexity.

Our study design had several strengths. We used a blinded, dual-panel approach, with both gastroenterologists and lay patients independently rating the same set of AI responses. This 2 × perspective (expert and consumer) provides a robust assessment of both technical quality and real-world usefulness. The set of 39 simulated patient questions covered six thematic domains (general GERD knowledge, symptoms/diagnosis, causes/risk factors, diet, treatments, lifestyle), ensuring that findings are not restricted to a single subtopic. We also systematically measured reading difficulty and structural characteristics of each answer, linking quantitative readability indices to subjective ratings. These multifaceted evaluations (comprehensiveness, accuracy, empathy, actionability, readability) offer a comprehensive picture of each model’s strengths and weaknesses.

Our study also had several limitations that merit consideration. All models were queried with a single fixed prompt (one-shot) per question, and we did not explore prompt engineering, few-shot examples, or multi-turn dialogues, each of which could improve LLM output. We also performed multiple pairwise comparisons without formal correction for multiplicity, which may have increased the risk of type I error. However, the observed differences were consistent in direction and magnitude across multiple domains and were concordant between physician and patient evaluations, supporting the overall interpretability of the findings. Finally, although all responses were independently rated using standardized, anchored scoring rubrics, formal inter-rater reliability metrics (such as intraclass correlation coefficients or kappa-based statistics) were not calculated.

The clinical implications of the study are substantial. First, LLMs can serve as valuable adjuncts for patient education and public health communication, improving understanding of gastrointestinal symptoms and promoting earlier health-seeking behavior. By providing accessible, conversational explanations, LLMs can bridge information gaps that often delay presentation in GERD and dyspepsia. Prior studies have shown that interactive AI tools can increase patient engagement and comprehension compared with static educational materials[23-25].

In this context, LLMs-through their detailed and empathetic outputs-may enhance patient reassurance and encourage appropriate medical consultation. Improved understanding of warning symptoms such as persistent heartburn or dysphagia, could in turn, facilitate earlier diagnosis of conditions like Barrett’s esophagus or peptic stricture. Thus, well-regulated LLM deployment could indirectly contribute to early detection and preventive healthcare engagement.

Second, LLMs may be integrated into clinical workflows as triage or counseling aids. For example, a physician could use Gemini or Claude to generate structured summaries or educational after-visit messages, saving time while maintaining empathy and personalization. Conversely, ChatGPT’s concise readability could be leveraged for routine discharge instructions or FAQs within electronic health portals. However, human oversight remains essential. Even when accuracy scores are high, occasional factual omissions or misinterpretations occur-echoing the cautionary notes raised in recent commentaries in JAMA Network Open and NEJM AI, which emphasize that AI outputs, although fluent, may propagate subtle inaccuracies if unverified[26,27].

Models that prioritize empathy and directive language (e.g., Claude-4, Gemini-2.5) may be suitable for supervised use in clinical counseling or chronic-disease education, while those emphasizing clarity (e.g., ChatGPT-5) may be better suited for general-public health messaging. A “human-in-the-loop” paradigm-in which AI drafts are reviewed, corrected, and tailored by clinicians-should remain the standard for deployment in gastroenterology.

CONCLUSION

In summary, our study provides one of the first head-to-head comparisons of multiple next-generation LLMs in a gastrointestinal context. Gemini-2.5 and Claude-4 produced the most clinically comprehensive, actionable, and empathetic responses, while ChatGPT-5 achieved superior readability and accessibility. This multidimensional trade-off underscores that no single model optimally balances accuracy, empathy, and simplicity. Future efforts should focus on model fine-tuning with gastroenterology-specific corpora, integration with guideline databases for retrieval-augmented grounding, and adaptive output formatting calibrated to patient literacy levels. Moreover, hybrid architectures-combining the clarity of ChatGPT-5 with the depth of Gemini-2.5 and the tone of Claude-4-may yield superior composite performance.

Ultimately, LLMs represent a transformative tool for patient engagement, health literacy, and early detection in gastroenterology. By translating complex medical information into interactive, comprehensible narratives, they can empower individuals to recognize symptoms earlier, seek medical attention promptly, and participate more actively in their care. As AI capabilities evolve, safe and ethically governed integration of LLMs into clinical and educational frameworks could substantially enhance both patient understanding and population-level health outcomes. Future developments could explore hybrid or ensemble methods that leverage the strengths of different models, for instance, using one model for generating medically comprehensive drafts and another for optimizing readability and tone.