Do bots provide correct and adequate guidance regarding acidity: A blinded comparison rated by patients and physicians

Table 1 Domains and scoring criteria for evaluating patient education responses

Domain	Scale	Description	Example anchors
Accuracy	1-4	Is the medical information factually correct, up-to-date, and aligned with current GERD guidelines?	1 = incorrect/misleading; 4 = entirely accurate/evidence-based
Comprehensiveness	1-4	Does the answer cover all important aspects of the question (causes, management, when to seek care) without omissions?	1 = very incomplete; 4 = fully comprehensive
Empathy/tone	1-4	Does the answer communicate in a patient-friendly, empathetic way that acknowledges concerns and emotions?	1 = cold/impersonal; 4 = very empathetic/reassuring
Actionability	1-5	How clear and useful are the next steps or advice for the patient (e.g., lifestyle changes, when to see a doctor)?	1 = no practical advice; 5 = very actionable with clear guidance

Each response was assessed across four domains (accuracy, comprehensiveness, empathy/tone, and actionability) using predefined ordinal scales with example anchors provided. GERD: Gastroesophageal reflux disease.

Table 2 Patient-rated domains and scoring criteria

Domain	Scale	Description	Example anchors
Comprehensibility	1-4	How easy is the answer to understand?	1 = very hard to understand, overly technical or confusing; 4 = very easy to understand, written in clear layman’s terms
Empathy	1-4	Did the tone feel caring and supportive?	1 = not at all empathetic; 4 = very empathetic and supportive in tone
Actionability	1-5	Is the advice useful and feasible? Would the reader know what to do next after reading?	1 = not actionable or no advice; 5 = extremely actionable with clear, helpful next steps

Responses were evaluated by patients for comprehensibility, empathy, and actionability using predefined ordinal scales, with illustrative anchors provided for each domain.

Table 3 Readability indices used to evaluate artificial intelligence-generated patient education responses

Index	Description	Scoring/interpretation	Ref.
FRE	Based on average sentence length and syllables per word, indicates text readability	0-100 scale; higher = easier; approximately 60 = 8th grade; approximately 30s = college-level
SMOG	Estimates the grade level required to understand the text, focusing on complex polysyllabic words	Reported as United States education grade (e.g., 8th, 12th)	[11]
ARI	Uses characters per word and words per sentence to estimate grade level required	Score corresponds to United States grade level	[12]
GFI	Estimates years of formal education needed to understand text, emphasizing sentence length and complex words	Range 6-17; 6-6.9 recommended for patient education	[13]
ARC	Composite measure averaging multiple indices (e.g., SMOG, ARI, GFI, FRE) to estimate grade level	Reflects approximate United States grade level needed for comprehension	-

Each measure provides an estimate of text difficulty, expressed as either a grade level or readability score. ARC: Average reading level consensus; ARI: Automated readability index; FRE: Flesch reading ease; GFI: Gunning fog index; SMOG: Simple Measure of Gobbledygook.

Table 4 Physician assessment of large language model responses to patient education questions on acidity

Characteristics	ChatGPT-5a	Gemini-2.5a	Claude-4a	P value
				A vs B	A vs C	B vs C
Overall
Comprehensiveness	2.86 ± 0.49	3.36 ± 0.52	3.43 ± 0.58	0.0001	0.0001	0.5763
Accuracy	2.89 ± 0.53	3.30 ± 0.51	3.34 ± 0.56	0.0008	0.0005	0.7425
Actionability	2.73 ± 0.62	3.67 ± 0.74	3.45 ± 0.53	0.0001	0.0001	0.1353
Empathy	2.64 ± 0.59	3.33 ± 0.57	3.38 ± 0.55	0.0001	0.0001	0.6945
Domain wise
Domain 1: General understanding
Comprehensiveness	2.55 ± 0.51	3.22 ± 0.43	3.33 ± 0.48	0.0001	0.0001	0.2898
Accuracy	2.66 ± 0.48	3.16 ± 0.51	3.43 ± 0.67	0.0001	0.0001	0.0488
Actionability	2.55 ± 0.51	3.39 ± 0.61	3.44 ± 0.51	0.0001	0.0001	0.6956
Empathy	2.61 ± 0.50	3.11 ± 0.67	3.33 ± 0.48	0.0004	0.0001	0.0996
Domain 2: Symptoms and diagnosis
Comprehensiveness	2.71 ± 0.46	3.43 ± 0.51	3.43 ± 0.67	0.0001	0.0001	1.0000
Accuracy	2.81 ± 0.51	3.28 ± 0.64	3.23 ± 0.62	0.0006	0.0016	0.7270
Actionability	2.76 ± 0.70	3.90 ± 0.70	3.47 ± 0.51	0.0001	0.0001	0.0027
Empathy	2.90 ± 0.53	3.42 ± 0.59	3.33 ± 0.57	0.0001	0.0009	0.4954
Domain 3: Causes and risk factors
Comprehensiveness	2.84 ± 0.42	3.56 ± 0.49	3.61 ± 0.50	0.0001	0.0001	0.6568
Accuracy	2.94 ± 0.23	3.44 ± 0.51	3.44 ± 0.51	0.0001	0.0001	1.0000
Actionability	2.83 ± 0.70	3.83 ± 0.98	3.50 ± 0.51	0.0001	0.0001	0.0660
Empathy	2.56 ± 0.61	3.33 ± 0.48	3.50 ± 0.51	0.0001	0.0001	0.1337
Domain 4: Dietary considerations
Comprehensiveness	2.83 ± 0.39	3.42 ± 0.51	3.50 ± 0.52	0.0001	0.0001	0.4948
Accuracy	2.75 ± 0.45	3.33 ± 0.49	3.41 ± 0.51	0.0001	0.0001	0.4821
Actionability	2.67 ± 0.49	3.67 ± 0.78	3.50 ± 0.53	0.0001	0.0001	0.2638
Empathy	2.50 ± 0.52	3.42 ± 0.51	3.47 ± 0.47	0.0001	0.0001	0.6538
Domain 5: Treatment and management
Comprehensiveness	2.90 ± 0.53	3.42 ± 0.47	3.52 ± 0.51	0.0001	0.0001	0.3707
Accuracy	3.05 ± 0.50	3.52 ± 0.51	3.42 ± 0.51	0.0001	0.0018	0.3893
Actionability	2.89 ± 0.48	3.67 ± 0.65	3.47 ± 0.51	0.0001	0.0001	0.1347
Empathy	2.71 ± 0.56	3.47 ± 0.52	3.47 ± 0.51	0.0001	0.0001	1.0000
Domain 6: Lifestyle factors
Comprehensiveness	3.11 ± 0.51	3.26 ± 0.59	3.29 ± 0.67	0.2334	0.1859	0.8343
Accuracy	3.03 ± 0.71	3.11 ± 0.69	3.22 ± 0.64	0.6153	0.2183	0.0001
Actionability	2.67 ± 0.68	3.59 ± 0.69	3.37 ± 0.63	0.0001	0.0001	0.1456
Empathy	2.52 ± 0.70	3.26 ± 0.59	3.26 ± 0.65	0.0001	0.0001	1.0000

^aP < 0.05.

P values indicate pairwise comparisons (ChatGPT-5 vs Gemini-2.5, ChatGPT-5 vs Claude-4a, Gemini-2.5 vs Claude-4a). Scores represent mean ± SD across three blinded gastroenterologists. Responses were evaluated for comprehensiveness, accuracy, actionability, and empathy using predefined ordinal scales (1-4 for comprehensiveness, accuracy, and empathy; 1-5 for actionability). Comparisons were made between ChatGPT-5, Gemini-2.5, and Claude-4 overall and across six thematic domains (general understanding, symptoms and diagnosis, causes and risk factors, dietary considerations, treatment and management, and lifestyle factors). A: ChatGPT-5; B: Gemini-2.5; C: Claude-4a.

Table 5 Physician categorical assessment of responses generated by ChatGPT-5, Gemini-2.5, and Claude-4 across four key domains: Comprehensiveness, accuracy, actionability, and empathy, n (%)

Characteristics	ChatGPT-5a	Gemini-2.5a	Claude-4a	P value
Comprehensiveness
Minimal	2 (5.12)	1 (2.6)	1 (2.6)	0.029
Moderately	32 (82.1)	27(69.2)	20 (51.3)
Highly	5 (12.8)	11(28.2)	18 (46.2)
Accuracy
Partially	5 (12.8)	1 (2.6)	1 (2.6)	0.013
Mostly	32 (82.1)	27(69.2)	26 (66.7)
Entirely accurate and evidence-based	2 (5.1)	11 (28.2)	12 (30.8)
Actionability
Slightly actionable	10 (25.5)	1 (2.6)	1 (2.6)	< 0.00001
Moderately actionable	26 (66.7)	14 (35.8)	20 (51.2)
Actionable	2 (5.2)	17 (43.7)	17 (43.6)
Highly actionable	1 (2.6)	7 (17.9)	1 (2.6)
Empathy
Minimally empathetic	17 (43.65)	1 (2.6)	1 (2.6)	0.00001
Moderately empathetic	20 (51.3)	26 (66.7)	25 (64.1)
Highly empathetic	2 (5.1)	12 (30.8)	13 (33.3)

^aP < 0.05.

P values indicate pairwise comparisons (ChatGPT-5 vs Gemini-2.5, ChatGPT-5 vs Claude-4a, Gemini-2.5 vs Claude-4a). P values were calculated using χ² tests to compare distributions between models.

Table 6 Patient assessment of large language model responses

Characteristics	ChatGPT-5a	Gemini-2.5a	Claude-4a	P value
				A vs B	A vs C	B vs C
Overall
Comprehensiveness	3.90 ± 0.29	3.94 ± 0.23	3.95 ± 0.21	0.5018	0.3859	0.8416
Actionability	3.41 ± 0.62	3.81 ± 0.40	3.86 ± 0.34	0.0001	0.0002	0.5538
Empathy	3.74 ± 0.46	3.89 ± 0.31	3.86 ± 0.35	0.0954	0.1941	0.6850
Domain wise
Domain 1: General understanding
Comprehensiveness	3.84 ± 0.36	3.95 ± 0.22	3.97 ± 0.18	0.1076	0.0472	0.6616
Actionability	3.83 ± 0.39	3.92 ± 0.27	3.75 ± 0.44	0.2397	0.3982	0.0432
Empathy	3.56 ± 0.55	3.87 ± 0.35	3.86 ± 0.35	0.0040	0.0053	0.8999
Domain 2: Symptoms and diagnosis
Comprehensiveness	3.93 ± 0.26	3.93 ± 0.25	3.94 ± 0.23	1.0000	0.8577	0.8546
Actionability	3.56 ± 0.61	3.88 ± 0.32	3.85 ± 0.36	0.0049	0.0126	0.6984
Empathy	3.75 ± 0.45	3.88 ± 0.32	3.86 ± 0.34	0.1456	0.2270	0.7898
Domain 3: Causes and risk factors
Comprehensiveness	3.89 ± 0.31	3.91 ± 0.29	3.95 ± 0.22	0.7694	0.3274	0.4946
Actionability	3.51 ± 0.61	3.81 ± 0.39	3.85 ± 0.21	0.0116	0.0015	0.5744
Empathy	3.77 ± 0.43	3.91 ± 0.29	3.90 ± 0.31	0.0960	0.1298	0.8834
Domain 4: Dietary considerations
Comprehensiveness	3.90 ± 0.31	3.97 ± 0.19	3.93 ± 0.27	0.2330	0.6499	0.4516
Actionability	3.13 ± 0.65	3.97 ± 0.16	3.89 ± 0.32	0.0001	0.0001	0.1667
Empathy	3.77 ± 0.46	3.92 ± 0.29	3.79 ± 0.44	0.0890	0.8450	0.1276
Domain 5: Treatment and management
Comprehensiveness	3.92 ± 0.26	3.95 ± 0.24	3.96 ± 0.18	0.5980	0.4320	0.8357
Actionability	3.25 ± 0.62	3.62 ± 0.49	3.86 ± 0.34	0.0046	0.0001	0.0141
Empathy	3.79 ± 0.43	3.88 ± 0.32	3.82 ± 0.41	0.2977	0.7534	0.4735
Domain 6: Lifestyle factors
Comprehensiveness	3.93 ± 0.26	3.96 ± 0.19	3.99 ± 0.17	0.5624	0.2315	0.4647
Actionability	3.22 ± 0.61	3.79 ± 0.41	3.84 ± 0.37	0.0001	0.0001	0.1788
Empathy	3.78 ± 0.43	3.91 ± 0.29	3.89 ± 0.31	0.1217	0.1989	0.7694

^aP < 0.05.

P values indicate pairwise comparisons (ChatGPT-5 vs Gemini-2.5, ChatGPT-5 vs Claude-4a, Gemini-2.5 vs Claude-4a). Patients evaluated responses from ChatGPT-5, Gemini-2.5, and Claude-4 across comprehensiveness, actionability, and empathy using Likert-type scales. Results are reported as the mean ± SD overall and stratified across six domains (general understanding, symptoms and diagnosis, causes and risk factors, dietary considerations, treatment and management, and lifestyle factors). A: ChatGPT-5; B: Gemini-2.5; C: Claude-4a.

Table 7 Categorical patient assessment of responses from ChatGPT-5, Gemini-2.5, and Claude-4

Characteristics	ChatGPT-5a	Gemini-2.5a	Claude-4a	P value
Comprehensiveness
Minimal	3 (7.7)	3 (7.7)	2 (5.1)	< 0.05
Moderately	29 (74.4)	24 (61.5)	23 (59.0)
Highly	7 (17.9)	13 (33.3)	14 (38.9)
Actionability
Slightly actionable	8 (22.2)	1 (2.6)	2 (5.1)	< 0.05
Moderately actionable	27 (69.2)	13 (33.3)	18 (46.2)
Actionable	2 (5.1)	18 (46.2)	16 (41.0)
Highly actionable	2 (5.1)	7 (17.9)	3 (7.7)
Empathy
Minimally empathetic	15 (38.5)	1 (2.6)	1 (2.6)	< 0.05
Moderately empathetic	21 (53.8)	23 59.0)	24 (61.5)
Highly empathetic	3 (8.3)	15 (38.5)	14 (35.9)

^aP < 0.05.

P values indicate pairwise comparisons (ChatGPT-5 vs Gemini-2.5, ChatGPT-5 vs Claude-4a, Gemini-2.5 vs Claude-4a). Patient ratings of comprehensiveness, actionability, and empathy are presented as n (%) across categorical levels (minimal, moderate, high). Comparisons between models were performed using χ² tests, with P < 0.05 indicating statistical significance.

Table 8 Readability assessment of responses generated by ChatGPT-5, Gemini-2.5, and Claude-4

Metric	ChatGPT-5			Gemini-2.5			Claude-4			P value
	Score	Grade	Reading difficulty	Score	Grade	Reading difficulty	Score	Grade	Reading difficulty	A vs B	A vs C	B vs C
ARC	12.6 ± 2.2	High school	Moderately difficult	17.4 ± 4.2	Graduate	Very difficult	19.6 ± 1.7	Doctorate	Extremely difficult	0.0001	0.0001	0.0033
ARI	13.5 ± 2.9	College	Fairly difficult	24.5 ± 9.2	Graduate	Very highly difficult	27.8 ± 2.9	Graduate	Extremely difficult	0.0001	0.0001	0.0359
GFI	13.1 ± 6.1	College freshman	Difficult	16. 9 ± 4.1	Early doctorate	Very difficult	18.8 ± 2.9	Doctorate	Very difficult	0.0018	0.0001	0.0207
FRE	34.7 ± 19.5	College level	Difficult	13.9 ± 17.2	Post-graduate	Very difficult	3.9 ± 3.3	Beyond post-graduate	Very difficult	0.0001	0.0001	0.0006
SMOG	10.1 ± 2.7	10th grade	Fairly difficult	16.2 ± 2.7	Graduate	Very highly difficult	18.8 ± 2.7	Doctorate level	Extremely difficult	0.0001	0.0001	0.0001

P < 0.05. P values indicate pairwise comparisons (ChatGPT-5 vs Gemini-2.5, ChatGPT-5 vs Claude-4a, Gemini-2.5 vs Claude-4a). Readability was evaluated using five established indices: Automated Readability Index (ARI), Gunning Fog Index (GFI), Flesch Reading Ease (FRE), Simple Measure of Gobbledygook (SMOG), and Average Reading Level Consensus (ARC). Results are presented as the mean ± SD along with corresponding United States grade levels and qualitative reading difficulty categories. Lower scores and grade levels indicate easier readability. A: ChatGPT-5; B: Gemini-2.5; C: Claude-4a.