Do bots provide correct and adequate guidance regarding acidity: A blinded comparison rated by patients and physicians

Figure 1 Physician assessment of comprehensiveness, accuracy, actionability, and empathy in responses generated by large language models. A: Physician evaluation of response comprehensiveness across large language models. Stacked bar chart showing categorical comprehensiveness ratings of ChatGPT-5, Gemini-2.5, and Claude-4. Responses were classified as minimal, moderately comprehensive, or highly comprehensive. Claude-4 and Gemini-2.5 produced a higher proportion of highly comprehensive responses compared with ChatGPT-5, consistent with χ² test results (P = 0.029); B: Physician evaluation of response accuracy across large language models. Stacked bar chart showing categorical accuracy ratings of ChatGPT-5, Gemini-2.5, and Claude-4 responses to patient education questions on acidity. Responses were classified as partially accurate, mostly accurate, or entirely accurate and evidence-based. Gemini-2.5 and Claude-4 demonstrated higher proportions of entirely accurate and evidence-based responses compared to ChatGPT-5, consistent with χ² test results (P = 0.013); C: Actionability of responses across ChatGPT-5, Gemini-2.5, and Claude-4. Distribution of actionability ratings for each model, categorized as slightly actionable, moderately actionable, actionable, and highly actionable. ChatGPT-5 responses were predominantly rated as moderately actionable, whereas Gemini-2.5 and Claude-4 had higher proportions of actionable and highly actionable responses. Statistical analyses showed significant differences across models (P < 0.00001); D: Physician’s assessment of empathy of responses generated by ChatGPT-5, Gemini-2.5, and Claude-4. Stacked bar chart showing the distribution of physician ratings across three empathy categories: Minimally empathetic, moderately empathetic, and highly empathetic. ChatGPT-5 responses were more often rated as minimally empathetic (43.6%), whereas Gemini-2.5 (66.7%) and Claude-4 (64.1%) were predominantly rated as moderately empathetic. Highly empathetic responses were observed more frequently with Gemini-2.5 (30.8%) and Claude-4 (33.3%) compared to ChatGPT-5 (5.1%). Differences across models were statistically significant (P < 0.00001).

Figure 2 Patient-rated comprehensiveness, actionability, and empathy of responses generated by large language models. A: Patient assessment of comprehensiveness of responses across large language models. The stacked bar chart represents the distribution of patient-perceived comprehensiveness (minimal, moderately, and highly comprehensive) in responses generated by ChatGPT-5, Gemini-2.5, and Claude-4. Percentages correspond to values shown in the accompanying table, with ChatGPT-5 responses being mostly rated as “moderately comprehensive”, while Gemini-2.5 and Claude-4 demonstrated a higher proportion of “highly comprehensive” responses (P < 0.05); B: Patient assessment of actionability of responses across large language models. The stacked bar chart shows the distribution of patient-rated actionability of responses generated by ChatGPT-5, Gemini-2.5, and Claude-4. Ratings ranged from “slight actionable” to “highly actionable”. ChatGPT-5 responses were most frequently rated as “moderately actionable”, whereas Gemini-2.5 and Claude-4 demonstrated higher proportions of “actionable” and “highly actionable” responses. Differences across models were statistically significant (P < 0.05); C: Patient assessment of empathy of responses generated by ChatGPT-5, Gemini-2.5, and Claude-4. Responses were categorized as minimally empathetic, moderately empathetic, or highly empathetic. ChatGPT-5 responses were more often rated as minimally empathetic (38.5%), whereas Gemini-2.5 (38.5%) and Claude-4 (35.9%) produced a greater proportion of highly empathetic responses. Differences across models were statistically significant (P < 0.05).