Large language models with supervised fine-tuning for inflammatory bowel disease diagnosis and prognosis

Abstract

BACKGROUND

Inflammatory bowel disease (IBD) is a chronic, incurable condition characterized by complex complications. Effective management of IBD demands extensive physician expertise and time, presenting major challenges due to limited experience and healthcare resources. Large language models (LLMs) may serve as efficient auxiliary tools to alleviate workforce pressures and enhance the accessibility and efficiency of healthcare services.

AIM

To evaluate the capabilities of an LLM trained by supervised fine-tuning (SFT) in the diagnosis and treatment of IBD.

METHODS

In this observational study, the diagnostic and treatment capabilities of a general LLM for IBD were evaluated using cases derived from real-world electronic health records. High-quality medical records were filtered for SFT to construct a dedicated IBD-LLM, followed by a direct head-to-head comparison of IBD care performance between the IBD-LLM and resident physicians.

RESULTS

The general LLM demonstrated suboptimal performance in IBD diagnosis and treatment, although its response consistency and harmlessness were confirmed. In the head-to-head comparison, IBD-LLM vs resident physicians achieved scores of 2.88 vs 3.47 out of 5 in diagnostic accuracy (P = 0.001), 4.18 vs 3.47 out of 5 in humanistic care (P = 0.001), 4.39 vs 3.41 out of 5 in clinical judgment (P = 0.001), 3.88 vs 3.53 out of 5 in communication skills (P = 0.002), and 4.11 vs 3.74 out of 5 in organizational effectiveness (P = 0.003). Although the IBD-LLM demonstrated performance comparable to resident physicians in three parameters, it showed significantly lower diagnostic accuracy. No safety risks or artificial intelligence hallucinations were observed in outputs from either the general LLM or IBD-LLM.

CONCLUSION

A specialized LLM can be efficiently trained with limited samples using SFT, and demonstrated significant potential for assisting with medical tasks under the supervision of attending physicians.

Key Words: Large language models; Inflammatory bowel disease; Disease screening; Supervised fine-tuning; Diagnosis

Core Tip: This study constructed a specialized inflammatory bowel disease-large language model (LLM) using supervised fine-tuning (SFT) on a limited number of high-quality data from real-world electronic health records. In a head-to-head comparison with resident physicians during simulated clinical interactions, the model excelled in humanistic care, clinical judgment, and organizational effectiveness, but demonstrated a significant deficit in diagnostic accuracy. Notably, no artificial intelligence hallucinations were observed, attributed to rigorous structured training and a conservative learning rate. These findings establish SFT-trained LLMs as valuable clinical adjuncts for enhancing patient interaction in resource-constrained settings, although complex diagnostic reasoning still mandates human oversight.

INTRODUCTION

Inflammatory bowel disease (IBD), which includes Crohn’s disease and ulcerative colitis, is characterized by a chronic, incurable course and complex progression[1-3]. Accurate diagnosis and effective treatment of IBD present significant challenges, requiring comprehensive medical history evaluation, precise diagnostic procedures, and individualized therapeutic adjustments[4-6]. Both developed and developing countries are facing the problem of a shortage of gastroenterologists[7,8]. Even the ratio of nurses is extremely low, with only 1.5 nurses available for every 250000 IBD patients[9]. However, millions of people worldwide suffer from IBD, and the incidence rate continues to rise, especially in newly industrialized regions[10,11]. The obvious mismatch between the increasing number of IBD patients and the limited clinical medical capacity highlights the urgent need for innovative solutions to bridge this gap.

Large language models (LLMs), as artificial intelligence (AI) systems, utilize neural network architectures and deep learning to represent complex relationships between words in textual datasets, aiming to mimic human capabilities for automated language analysis[12,13]. In the medical field, LLMs demonstrate significant application potential[14-17]. For instance, their ability to pass the United States medical licensing examination[15,18] indicates competent performance in static medical question-answering tasks. However, written examinations are unvalidated proxies for clinical competency[19], and the test performance of LLMs does not necessarily translate to real-world clinical effectiveness or utility[15,16]. The essence of clinical practice lies in dynamic physician-patient interactions, particularly in medical history-taking and diagnostic reasoning. Studies have revealed that LLMs exhibit markedly reduced performance in simulated hospital environments[20], highlighting their current limitations in dynamic information gathering and clinical decision-making. Autonomous deployment of LLMs for clinical decision-making or patient communication may thus introduce significant risks[21].

In real-world clinical practice, it is an open-ended question-and-answer situation without preset options. Medical decision-making depends on the accuracy and comprehensiveness of information collection. Currently, general LLMs find it difficult to meet the needs of complex clinical practices[22]. Structured and unstructured texts, such as electronic health records (EHRs)[23], clinical notes[24], and medical literature[25], can be applied to pre-train medical LLMs, enabling them to achieve excellent performance in medical tasks, offering significant potential to assist healthcare professionals in clinical decision-making and medical history collection across diverse scenarios[21,26]. However, training a specialized medical LLM from scratch remains prohibitively expensive and resource-intensive[22], requiring massive domain-specific datasets and computational power. Critically, supervised fine-tuning (SFT)[27-29] provides a cost-effective alternative by repurposing pre-trained general LLMs through targeted data adaptation, effectively extending their capabilities without full retraining.

To address this gap, our study leveraged SFT to efficiently construct an IBD-specific LLM with limited samples, overcoming the traditional sample-size barriers in medical AI development. Beyond model development, we rigorously evaluated its real-world applicability in IBD diagnosis and clinical history intake, aiming to bridge the translational divide between AI research and practical healthcare delivery.

MATERIALS AND METHODS

Data collection

This observational study was conducted and approved by the ethics committee of Beijing Tsinghua Changgung Hospital and all methods were performed in accordance with the relevant guidelines and regulations. The medical records used in this study were all sourced from Beijing Tsinghua Changgung Hospital, a tertiary hospital in China with 1500 beds. Furthermore, the Gastroenterology Department has appointed a dedicated case manager[30,31] to ensure the quality of medical records.

In this study, the medical records of 97 patients were selected as the training set. Of these, 63 cases were confirmed to have IBD and 34 cases had other diseases requiring differential diagnosis from IBD. Detailed information is shown in Table 1. Newly admitted IBD patients from November 2023 to November 2024 were selected as the validation set to ensure that the trained LLM had no prior exposure to this dataset in order to avoid bias.

Table 1 The distribution of disease types in the training set.

Disease name	Disease category	Number (n = 97)
Ulcerative colitis	IBD	22
Crohn’s disease	IBD	39
IBD-unclassified	IBD	2
Chronic diarrhea	Differential diagnosis	6
Functional diarrhea	Differential diagnosis	5
Food intolerance-associated diarrhea	Differential diagnosis	1
Infectious diarrhea	Differential diagnosis	2
Diarrhea, unspecified	Differential diagnosis	4
Functional abdominal pain	Differential diagnosis	6
Constipation	Differential diagnosis	8
Familial adenomatous polyposis	Differential diagnosis	2

IBD: Inflammatory bowel disease.

For the patients included in this study, an informed consent waiver was approved due to the use of retrospective data and the minimal risk posed to participants. All patient information was de-identified to protect privacy and maintain hospital information security.

Model training

In this study, we employed a decoder-only model architecture. Compared to encoder-only frameworks such as bidirectional encoder representations from transformers, decoder-only models enable direct generation of continuous natural language responses from input sequences thereby yielding enhanced fluency and coherence in long-form text generation. In contrast to encoder-decoder architectures (e.g., T5), decoder-only models exhibit more efficient capture of contextual dependencies, rendering them particularly well-suited for multi-turn dialogue scenarios.

For the model training process, we selected Doubao-32k-0615, a LLM developed by ByteDance, as the foundational model. Constructed on the Transformer Decoder architecture, Doubao boasts a context window of 32K tokens, demonstrating remarkable performance in long-text understanding and generation tasks. Following pre-training on a massive general corpus, Doubao has undergone validation across multiple professional domains, including healthcare, education, and finance.

SFT technique

SFT is a well-established and efficient technique for tailoring pre-trained language models to specific tasks by utilizing high-quality annotated datasets[32-35]. Through the construction of input-output pairs, such as symptom descriptions paired with corresponding diagnoses or medical questions with standardized responses, SFT enables targeted adjustment of model parameters. This approach significantly enhances performance on specific tasks while mitigating computational costs associated with full-scale retraining[33,36,37].

Following the de-identification of EHRs in the validation set, five gastroenterologists reconstructed and recast these records into doctor-patient dialogue formats. The objective was to authentically replicate dynamic clinical communication scenarios, ensuring the accuracy of clinical reasoning, the validity of medical judgments, and integrating the humanistic aspects of doctor-patient interaction.

To ensure that the quality of the dialogues could accurately reflect the diagnosis and treatment thinking and completely present the medical decision-making process of doctors after collecting and integrating information, three doctors with both professional backgrounds in clinical medicine and model learning rules reviewed and adjusted the rewritten medical dialogues again using the constructed high-quality medical corpus, and conducted SFT training on Doubao in the low-rank adaptation manner.

The training dataset consists of two components: The core dataset comprises 97 samples (with 2 × up sampling) of multi-turn dialogue data, meticulously annotated by medical experts and augmented to expand data diversity, comprehensively covering clinical scenarios such as symptom inquiry, differential diagnosis, and treatment recommendations for IBD. An auxiliary dataset of approximately 10000 general task samples was integrated to enhance the model’s generalization ability and language comprehension diversity. Through systematic grid search and cross-validation during parameter tuning, we determined the optimal parameter configuration: A learning rate of 0.0002 and 5 training epochs. The final training loss curve is presented in Figure 1. We trained Doubao into an upgraded version using the SFT technique and named it IBD-LLM. The process of constructing the medical corpus and training the model is shown in Figure 2.

Figure 1  The final training loss curve.

Figure 2 The process of constructing the medical corpus and training the model. EHR: Electronic health record; D-P: Doctor-patient; SFT: Supervised fine-tuning; IBD: Inflammatory bowel disease.

Evaluation

Ensuring the safety, ethical integrity, and lack of bias in LLMs for clinical applications is critical. We conducted a comprehensive evaluation of our proposed framework, focusing on patient safety and clinical utility. To ensure a fair comparison, the three resident physicians involved in the head-to-head evaluation were in their second to third year of gastroenterology residency training, each with at least one year of dedicated clinical experience in managing IBD under the supervision of attending gastroenterologists. This ensured that the residents possessed the fundamental clinical competency required for medical history collection and preliminary disease diagnosis.

Our unique evaluation framework was designed to: (1) Given that medical history collection is an essential skill for resident physicians and this task is mainly performed by resident physicians in clinical practice, we designed a head-to-head comparison between the SFT LLM (IBD-LLM) and the resident physicians in terms of their capabilities in medical history collection and disease diagnosis; and (2) Adopted a scoring system inspired by the mini-clinical evaluation exercise (mini-CEX)[38,39] to assess the interaction quality. This scoring system, modeled after the mini-CEX, evaluates key dimensions such as accuracy, completeness, professionalism, logical reasoning ability, and patient-centered care, ensuring a comprehensive and standardized assessment. The scoring dimensions were defined as follows: (1) Diagnostic accuracy: The correctness of the final diagnosis; (2) Humanistic care: Empathy, respect, and patient-centered communication; (3) Clinical judgment: The logical progression of inquiry, appropriateness of follow-up questions based on responses, and the ability to formulate a reasonable differential diagnosis process; (4) Communication skills: Clarity, professionalism, and efficiency of information delivery; and (5) Organizational effectiveness: Time management and logical structuring of the consultation. This approach highlights the IBD-LLM’s ability to perform comparably to resident physicians while maintaining high clinical standards.

Statistical analysis

To assess the general performance of the LLM in medical history collection and disease diagnosis, a 5-point Likert scale was employed (1 = unacceptable, 5 = excellent). Additionally, three attending physicians used a 3-point Likert scale (1 = unacceptable, 3 = excellent) to evaluate response consistency and the presence of offensive content. Doubao was approved to undergo SFT training after its outputs were confirmed to be stable and free of harmful information. The three attending physicians evaluated four interaction records for each patient in the validation set, three from the resident physicians and one from the IBD-LLM, without knowing the source of each record.

To enhance consistency and reliability, the attending physicians received standardized scoring training before the evaluation and conducted a simulation assessment on three cases selected from the training set. Additionally, after independent scoring, they held regular meetings to ensure consensus on the multidimensional scoring framework.

Statistical analysis was performed using GraphPad Prism 9.5. Normality tests were conducted on the data. Normally distributed measurement data are presented as mean ± SD. Comparisons between two groups were performed using the unpaired two-tailed Student’s t-test, while comparisons among multiple groups were analyzed using one-way analysis of variance followed by Tukey’s post hoc test. Non-normally distributed measurement data are presented as the median. Comparisons between two groups were performed using the Mann-Whitney U test, while comparisons among multiple groups were analyzed using the Kruskal-Wallis H test, supplemented by Dunn’s post hoc test. A P value < 0.05 was considered statistically significant.

RESULTS

Clinical characteristics

The clinical characteristics of IBD patients in the validation set are shown in Table 2, revealing the complexity and challenge in managing IBD patients. A significant proportion of patients showed low body mass index (BMI), with 26% having a BMI < 18.5 kg/m². Additionally, the cohort exhibited prolonged disease duration, as 100% of cases had a disease course ≥ 3 months and 36% ≥ 5 years. Notably, 30% of patients had a history of surgical intervention, and the majority presented with complex medication profiles. These characteristics fully reflect the complexity and refractory nature of IBD and pose extremely high demands on the diagnostic and therapeutic capabilities of clinicians. The multifaceted nature of IBD presents a stringent challenge for diagnostic tools. If LLM can effectively handle medical history collection and diagnosis for such a complex condition, it is poised to demonstrate comparable performance across other diseases.

Table 2 Clinical characteristics of inflammatory bowel disease patients in the validation set.

Clinical information	Total number of cases (n = 30)	Number of CD patients (n = 15)	Number of UC patients (n = 15)
Gender
Male	16	8	8
Female	14	7	7
BMI
< 18.5 kg/m2	8	5	3
≥ 18.5 kg/m2	22	10	12
Age
17-25 years	7	5	2
25-40 years	8	4	4
> 40 years	15	6	9
Disease duration
< 5 years	19	12	7
≥ 5 years	11	3	8
Disease activity
Remission	14	7	7
Active	16	8	8
Surgical history
Yes	9	9	0
No	21	6	15
Smoking and alcohol history
Yes	6	3	3
No	24	12	12
Comorbidities with other diseases
Yes	13	7	6
No	17	8	9
Number of treatment regimens
0	1	0	1
1	11	9	2
2 or more	18	6	12

CD: Crohn’s disease; UC: Ulcerative colitis; BMI: Body mass index.

Our validation set could cover a wide range of clinical application scenarios. The dataset featured a balanced distribution of Crohn’s disease (50%) and ulcerative colitis (50%), an equitable gender composition (53% male, 47% female), and a symmetrical representation of disease activity states (47% in remission, 53% active). It is worth noting that the treatment options for IBD are diverse and complex, and this validation set encompassed almost all current medical treatment options for IBD, including patients who had not yet been prescribed any medication at their initial visit.

General LLM evaluation results

Following the anonymization of EHRs in the validation set, a gastroenterologist assumed the role of a patient using EHR data to engage in interactions with Doubao. Interactions began with the patient’s chief complaint and concluded with a disease diagnosis. Three separate interactions were conducted for each EHR, and the experts systematically assessed the medical history collection and disease diagnosis processes documented in the interaction records. The overall process is shown in Figure 3.

Figure 3  Testing and scoring process of the general large language model.

The assessment results are shown in Figure 4. Doubao’s performance in medical history collection and disease diagnosis yielded the following scores: Poor (4%), unacceptable (92%), and good (4%). General LLMs demonstrate limited adaptability to real-world clinical scenarios. Doubao’s output consistency received a “good” rating across all evaluations (100%), and no instances of AI hallucinations or harmful information were detected during the output process (100%). The intraclass correlation coefficient (ICC) for inter-rater reliability was 0.725 [95% confidence interval (CI): 0.559-0.852], demonstrating a substantial level of consistency.

Figure 4 The assessment results of the Doubao. A: The ability in medical history collection and disease diagnosis; B: The ability in output stability; C: The ability of the appropriateness of the output content.

Head-to-head evaluation results

Following the anonymization of EHRs in the validation set, a gastroenterologist assumed the role of a patient using EHR data to engage in separate interactions with the IBD-LLM and resident physicians. The experts systematically evaluated the medical history collection and disease diagnosis processes documented in the interaction records. The detailed procedure is illustrated in Figure 5.

Figure 5 Head-to-head comparison between resident physicians and the inflammatory bowel disease-large language model. LLM: Large language model; SFT: Supervised fine-tuning; IBD: Inflammatory bowel disease.

In the head-to-head comparison between the IBD-LLM and resident physicians (Figure 6), the IBD-LLM achieved an average score of 19.44/25, while resident physicians scored an average of 17.67/25. The top-performing resident physician attained 20.40/25 in this evaluation. In terms of the scores for diagnostic accuracy, IBD-LLM and resident physicians (on average) scored 2.88 ± 0.49 vs 3.47 ± 0.59 (P = 0.001, out of 5 points), for humanistic care 4.18 ± 0.46 vs 3.47 ± 0.55 (P = 0.0001, out of 5 points), for clinical judgment 4.39 ± 0.76 vs 3.41 ± 1.12 (P = 0.0001, out of 5 points), for communication skills 3.88 ± 0.55 vs 3.53 (P = 0.002, out of 5 points), and for organizational effectiveness 4.11 ± 0.55 vs 3.74 ± 0.46 (P = 0.003, out of 5 points). The ICC for accuracy was 0.87 (95%CI: 0.81-0.92), indicating excellent consistency; for humanistic care, the ICC was 0.76 (95%CI: 0.67-0.83), reflecting good consistency; for clinical judgment, the ICC was 0.79 (95%CI: 0.71-0.85), with good consistency; for communication skills, the ICC was 0.83 (95%CI: 0.76-0.89), showing good consistency; and for organizational effectiveness, the ICC was 0.81 (95%CI: 0.73-0.87), also demonstrating good consistency.

Figure 6 Evaluated and averaged across three consultant-level gastroenterologists. A: Rating in five domains; B: Total scores (defined as the sum of domain-specific scores); C: Evaluated and averaged across three consultant-level gastroenterologists. LLM: Large language model; SFT: Supervised fine-tuning.

DISCUSSION

IBD shares overlapping symptoms with other gastrointestinal disorders[5], and its clinical manifestations are often variable and fluctuating[40], which complicates the diagnostic process. Misdiagnosis increases the risk of adverse events[41-43], such as anastomotic fistula, intra-abdominal abscess, and enterocutaneous fistula[44]. Approximately 40%-60% of patients exhibit primary non-response to pharmacological therapies[45-48], while 10%-40% develop secondary non-response or experience adverse events necessitating treatment discontinuation[49,50]. Complications frequently occur during the course of IBD, and intestinal surgery is sometimes inevitable[5,51]. Compared with the non-IBD group, the medical expenses of patients with IBD are significantly elevated[52,53]. The majority of the patients expressed concern over their financial circumstances[54], and about 25% of them fall into financial difficulties due to medical expenses[55]. The high cost of IBD is essentially due to the shortage of professionals, which leads to failure in receiving systematic diagnosis and treatment in the early stage of the disease, resulting in poor disease control[56-58].

LLMs suffer from substantial inaccuracy and uncertainty issues[21], while their constraints in dynamic interaction and iterative reasoning limit their applicability in real-world medical settings, failing to meet the demands of personalized patient care[59-61]. Our study demonstrates that an LLM fine-tuned with limited samples via SFT can achieve performance comparable to resident physicians across several key domains of IBD management, while excelling in humanistic care and organizational effectiveness. However, a critical finding was the IBD-LLM’s lower score in diagnostic accuracy compared to residents (2.88 vs 3.47, P = 0.001). This discrepancy may stem from the nature of the training data; while SFT effectively taught the model conversational patterns and clinical reasoning from expert-annotated dialogues, it may not have fully captured the nuanced, iterative diagnostic heuristics that experienced clinicians develop over years. Specifically, a detailed analysis of the lower-scoring interactions revealed that the IBD-LLM occasionally exhibited incomplete medical history taking in complex cases and omitted crucial differential diagnoses when symptoms overlapped with other gastrointestinal disorders. For instance, for concurrent abdominal pain and altered bowel habits, the IBD-LLM sometimes prematurely concluded without probing for alarm symptoms or recommending endoscopy, thus missing underlying IBD. Such errors highlight its limitations with ambiguous presentations, supporting its role as an adjunct rather than a standalone diagnostician. This suggests that while SFT can rapidly enhance a model’s utility, mastering complex diagnostic reasoning remains a distinct challenge that may require more extensive or specialized data.

The model’s superior performance in humanistic care and clinical judgment is noteworthy. This could be attributed to the high-quality dialogues used for SFT, which were meticulously reconstructed to incorporate not only clinical facts but also the empathetic and logical aspects of doctor-patient communication. This highlights a potential strength of the SFT approach: By curating data that exemplifies desired clinical behaviors, we can shape an AI’s “soft skills” more effectively than its “hard skills” like diagnosis. This finding confirms that AI-assisted tools represent viable solutions for healthcare systems grappling with physician shortages, particularly in resource-constrained settings, but their optimal role may be as an adjunct that enhances efficiency and patient interaction, rather than as a standalone diagnostician.

A notable finding was the absence of AI hallucinations during the simulated interactions. While this is encouraging, it is crucial to view this result within the context of our study’s limitations. The absence of hallucinations may be primarily attributed to the strict question-and-answer structure of the high-quality SFT corpus, which closely bound the model’s outputs to established clinical reasoning pathways. Additionally, the extremely low learning rate (0.0002) may have played a crucial constraining role, preventing the model from overfitting or generating unprompted associations. This does not guarantee similar safety in more complex, real-world settings, and the risk remains, particularly as the model is tasked with more ambiguous patient presentations. Therefore, future safeguards, such as real-time clinician validation and integration with evidence-based databases, are essential before clinical deployment.

Our decision to use a general LLM as a baseline was strategic, offering distinct advantages in resource accessibility and fine-tuning feasibility over domain-specific models. This approach effectively lowers the barrier for developing specialized medical AI, aligning with the practical needs of resource-limited healthcare environments. While our IBD-LLM shows promise, its current evaluation was conducted in a simulated setting. Real clinical practice is far more complex and dynamic, meaning the model’s performance may not fully align with its effectiveness in actual patient care. Prospective validation studies are necessary to establish its clinical viability. The study is further limited by its single-center data, simulated evaluation environment, lack of iterative learning mechanisms, and notably, the exceedingly limited sample sizes for both the training set (97 cases) and validation set (30 cases). These small sample sizes are insufficient to encompass the vast complexity of IBD and its wide array of differential diagnoses. Therefore, future research must prioritize large-scale, multicenter, prospective validation to confirm the model’s effectiveness and safety in real-world clinical workflows.

CONCLUSION

In this study, we demonstrated that a specialized IBD-LLM can be efficiently trained via SFT with a small, high-quality dataset. The model showed promising capabilities, performing on par with resident physicians in several critical domains of medical history collection and clinical interaction, which highlights its potential as a supportive tool to alleviate workforce pressures (Figure 7). However, its lower performance in diagnostic accuracy compared to residents underscores that while SFT can rapidly enhance a model’s clinical utility, mastering complex diagnostic reasoning remains a key challenge. This finding suggests that the model is best positioned as an adjunct to assist physicians, particularly in high-volume or resource-constrained settings, rather than as a replacement for clinical judgment. While the absence of AI hallucinations in our simulated study is a positive indicator, this result is conditional and does not eliminate the risk in broader clinical applications. The study is limited by its single-center data, simulated evaluation environment, and lack of iterative learning mechanisms. Therefore, future research must prioritize multicenter, prospective validation to confirm the model’s effectiveness and safety in real-world clinical workflows. Addressing these limitations through larger, more diverse datasets and continuous learning frameworks will be essential to fully realize the transformative potential of AI-assisted tools in medicine.

Figure 7 Clinical scenarios with and without inflammatory bowel disease-large language model assistance. LLM: Large language model; AI: Artificial intelligence; IBD: Inflammatory bowel disease.