Machine learning-based decision tool for selecting patients with idiopathic acute pancreatitis for endosonography to exclude a biliary aetiology

Abstract

BACKGROUND

Biliary microlithiasis/sludge is detected in approximately 30% of patients with idiopathic acute pancreatitis (IAP). As recurrent biliary pancreatitis can be prevented, the underlying aetiology of IAP should be established.

AIM

To develop a machine learning (ML) based decision tool for the use of endosonography (EUS) in pancreatitis patients to detect sludge and microlithiasis.

METHODS

We retrospectively used routinely recorded clinical and laboratory parameters of 218 consecutive patients with confirmed AP admitted to our tertiary care hospital between 2015 and 2020. Patients who did not receive EUS as part of the diagnostic work-up and whose pancreatitis episode could be adequately explained by other causes than biliary sludge and microlithiasis were excluded. We trained supervised ML classifiers using H₂O.ai automatically selecting the best suitable predictor model to predict microlithiasis/sludge. The predictor model was further validated in two independent retrospective cohorts from two tertiary care centers (117 patients).

RESULTS

Twenty-eight categorized patients’ variables recorded at admission were identified to compute the predictor model with an accuracy of 0.84 [95% confidence interval (CI): 0.791-0.9185], positive predictive value of 0.84, and negative predictive value of 0.80 in the identification cohort (218 patients). In the validation cohort, the robustness of the prediction model was confirmed with an accuracy of 0.76 (95%CI: 0.673-0.8347), positive predictive value of 0.76, and negative predictive value of 0.78 (117 patients).

CONCLUSION

We present a robust and validated ML-based predictor model consisting of routinely recorded parameters at admission that can predict biliary sludge and microlithiasis as the cause of AP.

Key Words: Acute pancreatitis; Idiopathic acute pancreatitis; Biliary pancreatitis; Microlithiasis; Sludge; Endosonography

Core Tip: Occult biliary lithiasis represents the largest monocausally treatable aetiology group within idiopathic acute pancreatitis cases. The identification of this subgroup protects patients from pancreatitis recurrences and over- or underdiagnosis. Based on 28 easy-to-collect and widely available patient variables, a machine learning-based prediction score can be used to predict the presence or absence of biliary sludge or microlithiasis in the context of pancreatitis hospitalisation. We provide a web-based prediction tool to select patients for endosonography to investigate microlithiasis or sludge as the cause of pancreatitis and treat them accordingly.

INTRODUCTION

Pancreatitis is a high incidence disease and the underlying cause for the highest number of patients admitted to hospital admission of all benign gastrointestinal-disorders[1]. In approximately 25% of patients with acute pancreatitis (AP), aetiology cannot be established during the first episode of pancreatitis[2,3]. If the aetiology of AP cannot be identified by history, laboratory chemistry, and imaging, it is classified as “idiopathic” [idiopathic AP (IAP)]. Unclassified or idiopathic pancreatitis represents the third largest group of pancreatitis and is therefore of great importance from both a medical and a socioeconomic point of view requiring thorough workup[3,4]. All efforts should be made to elucidate a treatable aetiology to prevent further episodes of AP. A recent meta-analysis has shown that biliary aetiology is the most common cause of idiopathic pancreatitis with a prevalence of 30%[5]. Specifically, in light of morbidity and mortality of AP, it is crucial to differentiate the potentially treatable aetiology of AP triggered by biliary sludge and microlithiasis from idiopathic or other causes of AP. Unfortunately, due to a lack of unifying definition of biliary sludge and microlithiasis, it is currently impossible to assess the risk of sludge and/or microlithiasis as the cause of AP. In the absence of clear evidence, guidelines suggest to treat those patients by cholecystectomy and maybe biliary sphincterotomy. In line, the diagnostic IAP workup requires excluding biliary microconcrements as it is believed that detection and concrement removal and/or cholecystectomy can prevent further episodes of pancreatitis in over 85% of cases[6,7]. To facilitate decision-making on whether the patient should be referred to endosonography (EUS) followed by endoscopic retrograde cholangiopancreatography (ERCP) or cholecystectomy, we developed a predictive tool using a machine learning (ML)-based approach to estimate the probability of the presence of biliary sludge and/or microlithiasis at the time of presentation to the emergency department. The ML tool, which is based on routine laboratory values, will help clinicians to enrich the likelihood to detect microlithiasis or sludge at admission on EUS and hereby reduce the number of EUS exams in presumed acute idiopathic pancreatitis.

MATERIALS AND METHODS

Study design

We retrospectively studied 1340 confirmed and hospitalized patient cases of AP treated at LMU University Hospital Munich (tertiary care hospital) between January 1, 2015 and October 1, 2020 (ICD-10 codes used: K85.00-K85.91). Patient cohorts with identical inclusion criteria from the University Hospital of the Technical University Munich and the University Medical Center Goettingen served as the validation cohort. The study was conducted in accordance with the updated STARD guideline of 2015[8].

Participants

Only patients meeting the diagnostic criteria for AP as set in the APA/IAP guidelines and adapted in the German S3-Guideline were enrolled in the analysis[9,10]. The first classifier used was whether patients received an EUS during their initial hospital stay, reducing the number of patients for further analysis to 360. The endosonographies were each performed by an experienced endoscopist. In the majority (79%) of pancreatitis stays, EUS was performed on days 1-3. Of the 360 patients with EUS, a total of 142 cases were excluded from further analysis due to incomplete records or missing coding. Two hundred and eighteen patient cases with AP and EUS were then further stratified into a cohort (47 patients) with no other cause of pancreatitis than endosonographically detected biliary microconcrements (biliary sludge/microlithiasis; detection of concrements in the common bile duct or gallbladder and common bile duct) and 171 patients with other causes of AP (Figure 1). In the two study groups [AP + EUS: 47 × microlithiasis vs 171 × non-microlithiasis (other cause); Supplementary Table 1], history, alcohol consumption, sonography, ERCP, or EUS findings, start or change of existing medication, known hereditary pancreatitis (available genetic testing of most prevalent susceptibility genes), and laboratory findings [lipase levels, immunoglobulin G subclasses, liver enzymes, triglycerides, and calcium level (corrected for blood serum albumin level)] were retrospectively evaluated. In the context of the laboratory value analyses, the values from the first blood analysis after admission of the respective patient stay was used in each case. The aim was to select patients in which microlithiasis/sludge was likely to subject them to EUS to reduce the number of EUS as an invasive, expensive procedure burdened with complications. To independently validate our machine-based algorithm, we obtained identical clinical data and inclusion criteria from two high volume German pancreas centers (University Hospital of the Technical University Munich: 22 × microlithiasis-AP, 51 × other-AP; University Medical Center Goettingen: 14 × microlithiasis-AP, 30 × other-AP; Supplementary Table 1). The definitions of the entities “biliary sludge” and “biliary microlithiasis” were taken from the endoscopic reports during the retrospective data evaluation and were not re-evaluated due to the current lack of an accepted unifying definition. Due to the differences between the participating centers in the use and partial equation of the two terms biliary sludge and microlithiasis, sludge-triggered pancreatitis was subsumed as biliary AP caused by microlithiasis.

Figure 1 Flow chart for development and external independent validation of microlithiasis prediction score. In the Ludwig-Maximilians-Universität in Munich identification cohort, 218 acute pancreatitis patients treated as inpatients between 2015-2020 were included in the final machine learning-based score survey. The validation cohort, consisting of 117 pancreatitis cases, was composed of patient data from the University Hospital of Göttingen and Technical University Munich. The microlithiasis predictive model was trained using data from both biliary sludge and biliary microlithiasis patients to cover the entirety of biliary microconcrements and to reflect the current lack of uniform definitions of biliary sludge and biliary microlithiasis in clinical practice. EUS: Endosonography; AP: Acute pancreatitis.

Test methods

All aspects of data reporting, predictive modeling, and validation reporting were performed in accordance with the TRIPOD guidelines[11]. A diagnostic reference standard for laboratory or imaging-based prediction of biliary sludge or microlithiasis in the context of AP has not yet been published. To derive the ML-based predictor model (index test), the following steps were performed (Figure 2): (1) Baseline variables (n = 192) were filtered leaving out variables with zero and near zero variance; (2) All numeric variables were classified into within limit, above upper limit, and below lower limit, based on clinical reference limits. All categorised variables were retained; (3) The training cohort was divided into a training (80%) and a test set (20%). Endpoint balancing was achieved by stratifying the classes by inducing the sampling rate of patients with microlithiasis and reducing the sampling rate of patients with other-AP. ML was performed based on all filtered baseline variables and data from the training set, resulting in a predictor based on all variables (base predictor model); and (4) To improve robustness and interpretability, low-impact variables were iteratively removed. An iterative predictive model with a reduced number of variables (n = 26) was obtained based on the performance in the test set.

Figure 2 Machine-learning based model for the prediction of biliary sludge and microlithiasis in the context of acute (presumed) idiopathic acute pancreatitis. Of the initial 192 variables analysed, 154 were included in the categorisation step after excluding those variables without evidence of variable variance. Using an auto-machine learning approach, the final (iterative) predictive model was developed via the base model step. ML: Machine learning.

All predictor models were constructed using the H₂O.ai platform (https://www.h2o.ai) selecting (with h2o.automl) the best suitable ML method in the training set. The parameters of each method were optimized by employing an internal ten-fold cross-validation on the training set. The optimal method was then applied to the test set to assess the final performance. In each loop, the best performing predictor model was identified from all predicted outcomes obtained using the performance measure logloss. Variables with a higher proportion of missing data (> 25% missing data) were also not excluded per se in order to base the final model on the broadest possible number of routinely available variables in the early phase of AP. The iterative predictive model obtained was externally validated in an independent retrospective dataset.

Statistical analysis

All data processing, modeling, and assessment of performances were done using R [version 4.0.4 (2021-02-15, “Lost Library Book”)] and visualized in R-studio (version 1.3.9.59). No unique algorithm was developed for this study. All data R scripts or functions are available online at the following link: https://github.com/mayerlelab/microlithiasisPredict. P values of < 0.05 were considered statistically significant if appropriate for the tests used.

RESULTS

Microlithiasis predictive score - results of the identification cohort

Between January 1, 2015 and October 1, 2020, 218 patients with AP received an EUS during their initial admission with AP at LMU University Hospital meeting the study inclusion criteria (Figure 1). In 47 of 218 pancreatitis patients, no causal pancreatitis aetiology other than endosonographically detected biliary microconcrements/sludge was found during the respective inpatient stay. Among 171 out of 218 pancreatitis patients with EUS, 52.6% (90/171) were classified as ‘idiopathic’, 21.6% (37/171) as acute on chronic, and 15.2% (27/171) with macrolithiasis as of biliary aetiology (Supplementary Table 1). Mean age in the microlithiasis/sludge cohort was 59.1 (SD 18.8) years in comparison to that of patients with AP of other aetiologies [54.6 (SD 17.1) years; P = 0.122]. Gender distribution was not statistically different in the two cohorts, with a male predominance in both cohorts [31/47 (66%) of microlithiasis patients and 103/171 (60.2%); P = 0.475] (Table 1). 76.6% of microlithiasis-AP patients were assessed as mild pancreatitis cases according to the revised Atlanta classification [36/47; 19.1% moderate (9/47) and 4.3% severe (2/47)]. In the other-AP cohort, 71.9% of patients were assessed as mild pancreatitis cases according to the revised Atlanta classification [123/171; 25.7% moderate (44/171) and 2.3% severe (4/171)]. A total of 29 variables from serum samples and 5 from urine were used to develop the ML-based microlithiasis prediction algorithm. All variables listed corresponded to the values measured at admission for each individual pancreatitis inpatient (see Table 1 for the list of variables used). To move from the base ML to the iterated ML model, weighting was done, taking scale variance into account. For the LMU identification cohort, age, triglycerides, sodium, glutamic pyruvic transaminase, erythrocytes, potassium, thyrotropin, protein (total), and leukocytes in descending order were of greatest importance in predicting microlithiasis/sludge. Using the iterated learner-based model, an accuracy of 0.8361 [95% confidence interval (CI): 0.791-0.9185; odds ratio = 20.88 (95%CI: 2.08-209.27)] with a sensitivity of 97.92% and positive predictive value (PPV) of 83.93% could be achieved for the prediction of microlithiasis as the trigger of pancreatitis [negative predictive value (NPV) = 0.80; specificity: 0.31; Table 2].

Table 1 Variables distribution at the Ludwig-Maximilians-Universität University Hospital Munich (identification cohort).

Variable	Microlithiasis (n = 47)	Other (n = 171)	Total (n = 218)	P value
Age (yr)				0.122
mean ± SD	59.1 ± 18.8	54.6 ± 17.1	55.6 ± 17.6
Range	30-92	24-90	24-92
Sex				0.474
Female	16/47 (34%)	68/171 (39.8%)	84/218 (38.5%)
Male	31/47 (66%)	103/171 (60.2%)	134/218 (61.5%)
Albumin				0.706
N-Miss	32/47 (68%)	90/171 (52.6%)	122/218 (55.9%)
LLN	2/47 (4.2%)	14/171 (8.1%)	16/218 (7.3%)
WL	13/47 (27.6%)	67/171 (39.1%)	80/218 (36.6%)
Alkaline phosphatase				0.667
N-Miss	1/47 (2.1%)	5/171 (2.9%)	6/218 (2.7%)
LLN	0/47 (0%)	1/171 (0.5%)	1/218 (0.4%)
ULN	22/47 (46.8%)	69/171 (40.3%)	91/218 (41.7%)
WL	24/47 (51.0%)	96/171 (56.1%)	120/218 (55.0%)
Total bilirubin				0.110
N-Miss	0/47 (0%)	2/171 (1.1%)	2/218 (0.9%)
ULN	23/47 (48.9%)	61/171 (35.6%)	84/218 (38.5%)
WL	24/47 (51.0%)	108/171 (63.1%)	132/218 (60.5%)
Calcium				0.033
N-Miss	20/47 (42.5%)	60/171 (35.0%)	80/218 (36.6%)
LLN	6/47 (12.7%)	7/171 (4.0%)	13/218 (5.9%)
ULN	0/47 (0%)	2/171 (1.1.%)	2/218 (0.9%)
WL	21/47 (44.6%)	102/171 (59.6%)	123/218 (56.4%)
Creatine kinase				0.073
N-Miss	29/47 (61.7%)	107/171 (62.5%)	136/218 (62.3%)
ULN	0/47 (0%)	10/171 (5.8%)	10/218 (4.5%)
WL	18/47 (38.2%)	54/171 (31.5%)	72/218 (33%)
CRP				0.391
ULN	37/47 (78.7%)	124/171 (72.5%)	161/218 (73.9%)
WL	10/47 (21.3%)	47/171 (27.5%)	57/218 (26.1%)
Total protein				0.743
N-Miss	30/47 (63.8%)	104/171 (60.8%)	134/218 (61.4%)
LLN	0/47 (0%)	1/171 (0.5%)	1/218 (0.4%)
ULN	3/47 (6.3%)	16/171 (9.3%)	19/218 (8.7%)
WL	14/47 (29.7%)	50/171 (29.2%)	64/218 (29.3%)
Erythrocytes				0.880
N-Miss	1/47 (2.1%)	0/171 (0%)	1/218 (0.4%)
LLN	14/47 (29.7%)	53/171 (30.9%)	67/218 (30.7%)
ULN	3/47 (6.3%)	8/171 (4.6%)	11/218 (5.0%)
WL	29/47 (61.7%)	110/171 (64.3%)	139/218 (63.7%)
Gamma-GT				0.108
N-Miss	0/47 (0%)	1/171 (0.5%)	1/218 (0.4%)
ULN	37/47 (78.7%)	113/171 (66%)	150/218 (68.8%)
WL	10/47 (21.3%)	57/171 (33.3%)	67/218 (30.7%)
AST/GOT				0.444
N-Miss	17/47 (36.1%)	51/171 (29.8%)	68/218 (31.1%)
ULN	21/47 (44.6%)	75/171 (43.8%)	96/218 (44.0%)
WL	9/47 (19.1%)	45/171 (26.3%)	54/218 (24.7%)
ALT/GPT				0.016
ULN	34/47 (72.3%)	90/171 (52.6%)	124/218 (56.9%)
WL	13/47 (27.7%)	81/171 (47.4%)	94/218 (43.1%)
Urea				0.429
N-Miss	31/47 (65.9%)	80/171 (46.7%)	111/218 (50.9%)
LLN	0/47 (0%)	7/171 (4%)	7/218 (3.2%)
ULN	3/47 (6.3%)	11/171 (6.4%)	14/218 (6.4%)
WL	13/47 (27.6%)	73/171 (42.6%)	86/218 (39.4%)
Hematocrit				0.304
N-Miss	1/47 (2.1%)	0/171 (0%)	1/218 (0.4%)
LLN	0/47 (0%)	2/171 (1.1%)	2/218 (0.9%)
ULN	41/47 (87.2%)	160/171 (93.5%)	201/218 (92.2%)
WL	5/47 (10.6%)	9/171 (5.2%)	14/218 (6.4%)
Haemoglobin				0.574
N-Miss	1/47 (2.1%)	0/171 (0%)	1/218 (0.4%)
LLN	12/47 (25.5%)	45/171 (26.3%)	57/218 (26.1%)
ULN	0/47 (0%)	4/171 (2.3%)	4/218 (1.8%)
WL	34/47 (72.3%)	122/171 (71.3%)	156/218 (71.5%)
INR				0.443
N-Miss	3/47 (6.3%)	9/171 (1.1%)	12/218 (5.5%)
ULN	8/47 (17.0%)	22/171 (12.8%)	30/218 (13.7%)
WL	36/47 (76.5%)	140/171 (81.8%)	176/218 (80.7%)
Potassium				0.270
N-Miss	9/47 (19.1%)	2/171 (1.1%)	11/218 (5%)
LLN	1/47 (2.1%)	7/171 (4%)	8/218 (3.6%)
ULN	0/47 (0%)	10/171 (5.8%)	10/218 (4.5%)
WL	37/47 (78.7%)	152/171 (88.8%)	189/218 (86.6%)
Serum creatinine				0.738
N-Miss	6/47 (12.7%)	0/171 (0%)	6/218 (2.7%)
LLN	1/47 (2.1%)	5/171 (2.9%)	6/218 (2.7%)
ULN	5/47 (10.6%)	29/171 (16.9%)	34/218 (15.5%)
WL	35/47 (74.4%)	137/171 (80.1%)	172/218 (78.8%)
LDH				0.020
N-Miss	7/47 (14.8%)	19/171 (11.1%)	26/218 (11.9%)
ULN	30/47 (63.8%)	83/171 (48.5%)	112/218 (51.8%)
WL	10/47 (21.2%)	69/171 (40.3%)	79/218 (36.2%)
Leukocytes				0.347
N-Miss	1/47 (2.1%)	0/171 (0%)	1/218 (0.4%)
LLN	1/47 (2.1%)	3/171 (1.7%)	4/218 (1.8%)
ULN	16/47 (34%)	80/171 (46.7%)	96/218 (44%)
WL	29/47 (61.7%)	88/171 (51.5%)	117/218 (53.6%)
Lipase				0.653
N-Miss	0/47 (0%)	1/171 (0.5%)	1/218 (0.4%)
LLN	0/47 (0%)	3/171 (1.7%)	3/218 (1.3%)
ULN	44/47 (93.6%)	157/171 (91.8%)	201/218 (92.2%)
WL	3/47 (6.4%)	10/171 (5.8%)	13/218 (5.9%)
MCH				0.498
N-Miss	1/47 (2.1%)	0/171 (0%)	1/218 (0.4%)
LLN	3/47 (6.3%)	20/171 (11.6%)	23/218 (10.5%)
ULN	4/47 (8.5%)	10/171 (5.8%)	14/218 (6.4%)
WL	39/47 (82.9%)	141/171 (82.4%)	180/218 (82.5%)
MCHC				0.108
N-Miss	1/47 (2.1%)	0/171 (0%)	1/218 (0.4%)
LLN	0/47 (0%)	13/171 (7.6%)	13/218 (5.9%)
ULN	1/47 (2.1%)	8/171 (4.6%)	9/218 (4.1%)
WL	45/47 (95.7%)	150/171 (87.7%)	195/218 (89.4%)
Triglycerides				0.004
N-Miss	27/47 (57.4%)	110/171 (64.3%)	137/218 (62.8%)
ULN	1/47 (2.1%)	24/171 (14%)	25/218 (11.4%)
WL	19/47 (40.4%)	37/171 (21.6%)	56/218 (25.6%)
RDW				0.329
N-Miss	5/47 (10.6%)	36/171 (21%)	41/218 (18.8%)
LLN	1/47 (2.1%)	11/171 (6.3%)	12/218 (5.5%)
ULN	5/47 (10.6%)	21/171 (12.2%)	26/218 (11.9%)
WL	36/47 (76.5%)	103/171 (60.2%)	139/218 (63.7%)
MCV				0.893
N-Miss	1/47 (2.1%)	0/171 (0%)	1/218 (0.4%)
LLN	4/47 (8.5%)	13/171 (7.6%)	17/218 (7.7%)
ULN	4/47 (8.5%)	12/171 (7.0%)	16/218 (7.3%)
WL	38/47 (80.8%)	146/171 (85.3%)	184/218 (84.4%)
Sodium				0.020
N-Miss	8/47 (17.0%)	1/171 (0.5%)	9/218 (4.1%)
LLN	1/47 (2.1%)	29/171 (16.9%)	30/218 (13.7%)
WL	38/47 (80.8%)	141/171 (82.4%)	179/218 (82.1%)
Quick’s value				0.479
N-Miss	3/47 (6.3%)	7/171 (4%)	10/218 (4.5%)
LLN	8/47 (17%)	23/171 (13.4%)	31/218 (14.2%)
ULN	20/47 (42.5%)	65/171 (38%)	85/218 (38.9%)
WL	16/47 (34%)	76/171 (44.4%)	92/218 (42.2%)
Thrombocytes				0.434
N-Miss	1/47 (2.1%)	0/171 (0%)	1/218 (0.4%)
LLN	8/47 (17%)	22/171 (12.8%)	30/218 (13.7%)
ULN	4/47 (8.5%)	26/171 (15.2%)	30/218 (13.7%)
WL	34/47 (72.3%)	123/171 (71.9%)	157/218 (72%)
TSH				0.567
N-Miss	27/47 (57.4%)	118/171 (69%)	145/218 (66.5%)
LLN	2/47 (4.2%)	4/171 (2.3%)	6/218 (2.7%)
ULN	4/47 (8.5%)	6/171 (3.5%)	10/218 (4.5%)
WL	14/47 (29.7%)	43/171 (25.1%)	57/218 (26.1%)
Bilirubin-urine				0.027
N-Miss	26/47 (55.3%)	102/171 (59.6%)	128/218 (58.7%)
Normal	8/47 (17%)	45/171 (26.3%)	53/218 (24.3%)
Abnormal	13/47 (27.6%)	24/171 (14.0%)	37/218 (16.9%)
Total protein-urine				0.231
N-Miss	26/47 (55.3%)	102/171 (59.6%)	128/218 (58.7%)
Normal	10/47 (21.3%)	43/171 (25.1%)	53/218 (24.3%)
Abnormal	11/47 (23.4%)	26/171 (15.2%)	37/218 (16.9%)
Ketones-urine				0.020
N-Miss	26/47 (55.3%)	106/171 (61.9%)	132/218 (60.5%)
Normal	21/47 (44.6%)	51/171 (29.8%)	72/218 (33%)
Abnormal	0/47 (0%)	14/171 (8.1%)	14/218 (6.4%)
Leukocytes-urine				0.162
N-Miss	26/47 (55.3%)	102/171 (59.6%)	128/218 (58.7%)
Normal	7/47 (14.8%)	35/171 (20.4%)	42/218 (19.2%)
Abnormal	14/47 (29.7%)	34/171 (19.8%)	48/218 (22%)
Specific gravity-urine				0.918
N-Miss	29/47 (61.7%)	113/171 (66%)	142/218 (65.1%)
mean ± SD	1018.33 ± 5.941	1018.103 ± 8.777	1018.158 ± 8.159
Range	1005.000-1025.000	1005.000-1030.000	1005.000-1030.000

All variables that were used for the final predictive model are listed. For laboratory or urine values, the first available value during the inpatient stay was used. For the investigated groups of microlithiasis-induced acute pancreatitis (n = 47) vs the group of pancreatitis induced by other aetiologies (n = 171), the variables were categorised as whether collected or not (N-Miss), in the case of laboratory values whether below the lower limit value, within the limit values, or above the upper limit value. For urine values, in addition to the rate of missing variables (N-Miss), it was categorised whether normal or abnormal. For the P value calculation using χ² test, variables with missing data shares of > 25% were not excluded. LLN: Lower limit value; WL: Within the limit value; ULN: Upper limit value; CRP: C-reactive protein; Gamma-GT: Gamma-glutamyl transpeptidase; AST: Aspartate aminotransferase; GOT: Glutamic oxalacetic transaminases; ALT: Alanine transaminase; GPT: Glutamic pyruvic transaminase; INR: International normalized ratio; LDH: Lactate dehydrogenase; MCH: Mean corpuscular hemoglobin; MCHC: Mean corpuscular hemoglobin concentration; RDW: Red blood cell distribution width; MCV: Mean corpuscular volume; TSH: Thyrotropin.

Table 2 Performance matrix (identification cohort vs validation cohort).

Accuracy	Sensitivity	Specificity	PPV	NPV
ID: 0.8361; 95%CI: 0.7191-0.9185	ID: 0.9792	ID: 0.3077	ID: 0.8393	ID: 0.800
VD: 0.7607; 95%CI: 0.673-0.8347	VD: 0.9630	VD: 0.3056	VD: 0.7573	VD: 0.7857

The performance values for the auto-machine-learning-based predictive model are listed. The identification cohort of the Ludwig-Maximilians-Universität in Munich University Hospital (ID) vs the data of the predictive models from the validation cohort with patient data from the University Hospital Göttingen and the Klinikum Rechts der Isar (VD). CI: Confidence interval; PPV: Positive predictive value; NPV: Negative predictive value.

Microlithiasis predictive score - results of the validation cohort

Data from two large-volume university pancreas centers were used for score validation. In total, a validation cohort of 36 patients with microlithiasis and 81 non-microlithiasis AP patients were retrieved from the clinical database at the University Hospital of the Technical University Munich (22 × microlithiasis-AP, 51 × other-AP) as well as the University Hospital Göttingen (14 × microlithiasis-AP, 30 × other-AP; Figure 1 and Table 3). In the Technical University Munich cohort, the group of other-AP patients was mainly alcohol-related [31/51 (60.8%)], while in the Göttingen cohort biliary macrolithiasis was held responsible for the majority of AP patients [16/33 (53.3%)]. Idiopathic aetiology was named as the second most frequent aetiology group in both external cohorts with about 30% each [Technical University Munich 17/51 (33.3%), Göttingen 10/33 (33.3%)] (Supplementary Table 1). Microlithiasis patients in the validation cohort were on average 60.1 (SD 18.4) years old, while patients from the other-AP cohort were 55.3 (SD 16.8) years old. In both groups (microlithiasis + other-AP), the majority of patients were male [24/36 (66.7%) and 46/81 (56.8%), respectively], resembling the identification cohort. 63.9% of microlithiasis-AP patients were assessed as mild pancreatitis cases according to the revised Atlanta classification [23/36; 27.7% moderate (10/36) and 8.3% severe (3/36)]. In the other-AP cohort, 59.2% of patients were assessed as mild pancreatitis cases according to the revised Atlanta classification [48/81; 27.2% moderate (22/81) and 12.5% severe (11/81)]. Using automated ML, the best-fitting model for iterative reduction of variables was used to achieve external validation of the microlithiasis predictive score using the optimized iterative ML model. For the validation cohort, based on the variables ordered by scaled importance in Figure 3, an accuracy of 0.7607 (95%CI: 0.673-0.8347), PPV of 0.7573, and NPV of 0.7857 were achieved (sensitivity: 0.96, specificity: 0.31; Table 2). The robustness of the model is shown in the alluvial plot in Figure 3, with only 3 out of 81 patients being misclassified as having microlithiasis and not as having other-AP, corresponding to the discretely higher NPV (compared to the PPV) in the validation cohort (Table 2).

Figure 3 Graphical representation of the prediction model variables according to importance of scale. A: Variables of the final (iterated) auto-machine learning prediction model are ordered by scale of importance; B and C: Precoat diagram showing robust positive and negative prediction (3/81 patient cases were misclassified as microlithiasis and not other-acute pancreatitis). Gamma-GT: Gamma-glutamyl transpeptidase; AST: Aspartate aminotransferase; GOT: Glutamic oxalacetic transaminases; ALT: Alanine transaminase; GPT: Glutamic pyruvic transaminase; LDH: Lactate dehydrogenase; RDW: Red blood cell distribution width.

Table 3 Variable distribution in the validation cohort (UMG + Technical University Munich).

Variable	Microlithiasis (n = 36)	Other (n = 81)	Total (n = 117)	P value
Age (yr)				0.162
mean ± SD	60.1 ± 18.4	55.3 ± 16.8	56.8 ± 17.4
Range	23-93	21-87	21-93
Sex				0.315
Female	12/36 (33.3%)	35/81 (43.2%)	47/117 (40.2%)
Male	24/36 (66.7%)	46/81 (56.8%)	70/117 (59.8%)
Alkaline phosphatase				0.032
N-Miss	1/36 (2.7%)	12/81 (14.8%)	13/117 (11.1%)
ULN	23/36 (63.8%)	30/81 (37%)	53/117 (45.2%)
WL	12/36 (33.2%)	39/81 (48.1%)	51/117 (43.5%)
Total bilirubin				0.003
ULN	21/36 (58.3%)	24/81 (29.6%)	45/117 (38.5%)
WL	15/36 (41.7%)	57/81 (70.4%)	72/117 (61.5%)
Creatine kinase				0.498
N-Miss	8/36 (22.2%)	32/81 (39.5%)	40/117 (34.1%)
ULN	5/36 (13.8%)	6/81 (7.4%)	11/117 (9.4%)
WL	23/36 (63.8%)	43/81 (53%)	66/117 (56.4%)
CRP				0.199
ULN	32/36 (88.9%)	64/81 (79%)	96/117 (88.8%)
WL	4/36 (11.1%)	17/81 (21%)	21/117 (17.9%)
Total protein				0.405
N-Miss	31/36 (86.1%)	73/81 (90.1%)	104/117 (88.8%)
WL	5/36 (13.8%)	8/81 (9.8%)	13/117 (11.1%)
Erythrocytes				0.650
N-Miss	0/36 (0%)	1/81 (1.2%)	1/117 (0.8%)
LLN	10/36 (27.7%)	25/81 (30.8%)	35/117 (29.9%)
ULN	4/36 (11.1%)	5/81 (6.1%)	9/117 (7.6%)
WL	22/36 (61.1%)	50/81 (61.7%)	72/117 (61.5%)
Gamma-GT				0.082
N-Miss	0/36 (0%)	2/81 (2.4%)	2/117 (1.7%)
ULN	32/36 (88.8%)	59/81 (72.8%)	91/117 (77.7%)
WL	4/36 (11.1%)	20/81 (24.6%)	24/117 (20.5%)
AST/GOT				0.079
N-Miss	0/36 (0%)	4/81 (4.9%)	4/117 (3.4%)
ULN	28/36 (77.8%)	47/81 (58%)	75/117 (64.1%)
WL	8/36 (22.2%)	30/81 (37%)	38/117 (32.4%)
ALT/GPT				0.052
N-Miss	0/36 (0%)	2/81 (2.4%)	2/117 (1.7%)
ULN	28/36 (77.8%)	43/81 (53%)	71/117 (60.6%)
WL	8/36 (22.2%)	30/81 (37%)	38/117 (32.4%)
Urea
N-Miss	7/36 (19.4%)	20/81 (24.6%)	27/117 (23%)
LLN	13/36 (36.1%)	41/81 (50.6%)	54/117 (46.1%)
ULN	0/36 (0%)	1/81 (1.2%)	1/117 (0.8%)
WL	16/36 (44.4%)	19/81 (23.4%)	35/117 (29.9%)
Hematocrit				< 0.001
N-Miss	0/36 (0%)	1/81 (1.2%)	1/117 (0.8%)
ULN	36/36 (100%)	80/81 (98.7%)	116/117 (99.1%)
Haemoglobin				0.725
N-Miss	0/36 (0%)	1/81 (1.2%)	1/117 (0.8%)
LLN	9/36 (25%)	18/81 (22.2%)	27/117 (23%)
ULN	3/36 (8.3%)	4/81 (4.9%)	7/117 (5.9%)
WL	32/36 (88.9%)	64/81 (79.0%)	96/117 (82%)
INR				0.440
ULN	2/36 (5.5%)	8/81 (9.8%)	10/117 (8.5%)
WL	34/36 (94.4%)	73/81 (90.1%)	107/117 (91.4%)
Potassium				0.985
LLN	2/36 (5.5%)	4/81 (4.9%)	6/117 (5.1%)
ULN	1/36 (2.7%)	2/81 (2.4%)	3/117 (2.5%)
WL	33/36 (91.6%)	75/81 (92.5%)	108/117 (92.3%)
Serum creatinine				0.909
LLN	2/36 (5.5%)	6/81 (7.4%)	8/117 (6.8%)
ULN	7/36 (19.4%)	14/81 (17.2%)	21/117 (17.9%)
WL	27/36 (75%)	61/81 (75.3%)	88/117 (75.2%)
LDH				0.018
N-Miss	6/36 (16.6%)	28/81 (34.5%)	34/117 (29%)
ULN	26/36 (72.2%)	33/81 (40.7%)	59/117 (50.4%)
WL	4/36 (11.1%)	20/81 (24.6%)	24/117 (20.5%)
Leukocytes				0.143
N-Miss	0/36 (0%)	1/81 (1.2%)	1/117 (0.8%)
LLN	1/36 (2.7%)	0/81 (0%)	1/117 (0.8%)
ULN	16/36 (44.4%)	47/81 (58%)	63/117 (53.8%)
WL	19/36 (52.7%)	33/81 (40.7%)	52/117 (44.4%)
Lipase				0.237
N-Miss	0/36 (0%)	2/81 (2.4%)	2/117 (1.7%)
ULN	32/36 (88.9%)	75/81 (92.5%)	107/117 (91.4%)
WL	4/36 (11.1%)	4/81 (4.9%)	8/117 (6.8%)
MCV				0.315
N-Miss	0/36 (0%)	1/81 (1.2%)	1/117 (0.8%)
LLN	3/36 (8.3%)	7/81 (8.6%)	10/117 (8.5%)
ULN	1/36 (2.7%)	9/81 (11.1%)	10/117 (8.5%)
WL	32/36 (88.9%)	64/81 (79%)	96/117 (82%)
Triglycerides				0.582
N-Miss	26/36 (72.2%)	43/81 (53%)	69/117 (58.9%)
ULN	3/36 (8.3%)	15/81 (18.5%)	18/117 (15.3%)
WL	7/36 (19.4%)	23/81 (28.3%)	30/117 (25.6%)
RDW
N-Miss	36/36 (100%)	81/81 (100%)	117/117 (100%)
False	0/36 (0%)	0/81 (0%)	0/117 (0%)
True	0/36 (0%)	0/81 (0%)	0/117 (0%)
Sodium				0.154
LLN	2/36 (5.5%)	12/81 (14.8%)	14/117 (11.9%)
WL	34/36 (94.4%)	69/81 (85.1%)	103/117 (88%)
Quick’s value				0.130
LLN	2/36 (5.5%)	6/81 (7.4%)	8/117 (6.8%)
ULN	13/36 (36.1%)	44/81 (54.3%)	57/117 (48.7%)
WL	21/36 (58.3%)	31/81 (38.2%)	52/117 (44.4%)
Thrombocytes				0.627
N-Miss	22/36 (61.1%)	51/81 (62.9%)	73/117 (62.3%)
LLN	2/36 (5.5%)	3/81 (3.7%)	5/117 (4.2%)
ULN	2/36 (5.5%)	2/81 (2.4%)	4/117 (3.4%)
WL	10/36 (27.7%)	25/81 (30.8%)	35/117 (29.9%)
TSH				0.773
N-Miss	6/36 (16.6%)	13/81 (16%)	19/117 (16.2%)
LLN	0/36 (0%)	1/81 (1.2%)	1/117 (0.8%)
ULN	1/36 (2.7%)	3/81 (3.7%)	4/117 (3.4%)
WL	29/36 (80.5%)	64/81 (79%)	93/117 (79.4%)

All variables that were used for the final predictive model are listed. For laboratory or urine values, the first available value during the inpatient stay was used. For the investigated groups of microlithiasis-induced acute pancreatitis (n = 36) vs the group of pancreatitis induced by other aetiologies (n = 81), the variables were categorised as whether collected or not (N-Miss), in the case of laboratory values whether below the lower limit value, within the limit values, or above the upper limit value. For the P value calculation using χ² test, variables with missing data shares of > 25% were not excluded. LLN: Lower limit value; WL: Within the limit value; ULN: Upper limit value; CRP: C-reactive protein; Gamma-GT: Gamma-glutamyl transpeptidase; AST: Aspartate aminotransferase; GOT: Glutamic oxalacetic transaminases; ALT: Alanine transaminase; GPT: Glutamic pyruvic transaminase; INR: International normalized ratio; LDH: Lactate dehydrogenase; MCH: Mean corpuscular hemoglobin; MCHC: Mean corpuscular hemoglobin concentration; RDW: Red blood cell distribution width; MCV: Mean corpuscular volume; TSH: Thyrotropin.

DISCUSSION

Previous and more recent studies on idiopathic pancreatitis still report a proportion of idiopathic pancreatitis stably at 20%-30%[12,13]. However, it has been suspected for decades and is increasingly supported by evidence that a large proportion of pancreatitis patients classified primarily as idiopathic actually suffer from a biliary aetiology and that detecting these patients during the first episode of pancreatitis is restricted due to the lack of availability of timely and high quality EUS exams[14]. Furthermore, there is a lack of reliable data on when, during an inpatient stay of an IAP-labeled patient, an EUS could detect biliary microconcrements as the trigger for pancreatitis without causing an unnecessary burden for the patient through overdiagnosis. This is an important question as we know from Oría et al[15] that common bile duct stones usually pass within 48 h, suggesting that microconcrements might even pass more rapidly and might not be detected on EUS. Prospective study data showed a corresponding variance of EUS-based biliary concrement detection rate of 19% in the low risk group, but 58% in the moderate risk group and 50% in the high risk group (grouping according to ASGE recommendation[16]). Risk stratification in terms of pre-test probability for EUS use < 48 h after hospital admission to rule in or out the presence of biliary concrements is warranted before intervention to overcome the lack of availability and reduce costs and side effects[12]. Diagnostic evaluation is complicated by the fact that biliary microconcrements could be a coincidental finding in the context of pancreatitis-induced gallbladder hypomotility, and therefore must always be understood in the individual patient’s setting, taking into account a PPV of a biliary pancreatitis origin greater than 85% with elevation of the alanine transaminase (ALT) above three times the upper limit of normal[17]. However, no causally effective drug for pancreatitis therapy is available in 2023 and the detection of causally remediable pancreatitis causes such as biliary microlithiasis or sludge will continue to play a decisive role in the prevention of further pancreatitis attacks. The efficacy of cholecystectomy in the cohort of IAP patients was shown in a meta-analysis with a recurrence rate of 11% compared to 38.9% in conservatively treated patients (risk ratio = 0.41; 95%CI: 0.16-1.07)[18]. Our ML-based approach of predicting biliary microlithiasis and sludge should therefore be understood as an approach to make up for the lack of evidence from prospective studies on the optimal timing of EUS in IAP patients as this score is based on widely available laboratory values and can be used to determine the probability of the presence of biliary microconcrements at admission. Our score helps to select patients for EUS with a high sensitivity and very high NPV and thus will reduce costs and complications of unnecessary EUS exams as well as allow to subject patients to further treatment to prevent recurrence of biliary pancreatitis at the time of presentation in the emergency department. Preliminary work on ML-based algorithms and prediction models in the context of AP has focused on severity assessment and prediction of complications[15]. A multicenter retrospective study used an auto-ML-based approach to predict pancreatitis severity, comparable to our ML approach, and achieved an area under the curve (AUC) of > 0.90 in the GBM model with a specificity and accuracy of both > 0.95 in the early detection of patients with a subsequently severe course of pancreatitis[19], outperforming clinically established non-ML-based scoring systems such as BISAP or Ranson underlying the relevance of ML approach over an educated guess[20,21]. ML-based prediction scores with regard to biliary microconcrements have not yet been published. Non-ML-based multivariate logistic regression models using widely available laboratory values have previously shown that an ALT level more than three times above the norm at patients’ admission [specificity of 82%, sensitivity of 60%, receiver operating characteristic (ROC)-AUC 0.733; P < 0.001] and age > 69.5 years (specificity 92%, sensitivity 57%, ROC-AUC 0.759; P < 0.001) act as the best predictors of biliary aetiology[17,22]. Here, our ML-based prediction score achieves a higher sensitivity (96.30%), whereby ALT and, above all, age also rank 4^th and 1^st in the weighting of our score, thus confirming the existing evidence in the area of non-ML laboratory value-based prediction of biliary aetiology of pancreatitis (Figure 3). Contrary to previously published studies on laboratory-based prediction of biliary pancreatitis aetiology, our prediction tool is based specifically on microlithiasis and sludge and not primarily on gallstones and occult microlithiasis/sludge subsumed in this cohort.

Our study has several limitations. First, the retrospective study approach did not allow us to generate a uniform definition of the two entities microlithiasis and sludge. Even after extensive literature research, we were unable to delineate a uniform but distinct definition of biliary microlithiasis and sludge. We thus decided to use the terms as synonyms between the endoscopy centers of the three participating university hospitals. This might impose a significant bias. The macrolithiasis, which was again clearly listed in the endoscopy findings across the universities, ensured quality of EUS. Likewise, the patient cohort declared as other-AP in terms of aetiology varied greatly between the participating centers (Supplementary Table 1). Ultimately, this probably reflects the individual diagnostic scope and the question of whether EUS can generate added value in the context of the individual patient. Also, due to the retrospective study design, no attempt could be made to increase the degree of purity of biliary (microlithiasis and sludge) triggered pancreatitis by uniformly fulfilling laboratory chemistry tests prior to EUS. This resulted in a proportion of patients of 36.6% with, for example, missing calcium values in the laboratory chemistry pancreatitis workup.

Our study is convincing in presenting for the first time a robust ML-based and externally validated prediction model for pancreatitis patients declared idiopathic early in the diagnostic workup and may be helpful as a noninvasive decision tool by combining simple and widely used laboratory values to decide for or against EUS. In order to make the microlithiasis predictive score and the relatively high number of underlying variables usable, a user-friendly interface is available online at the following link for use in the research context: https://github.com/mayerlelab/microlithiasisPredict. To illustrate the performance of the microlithiasis predictive score, we designed a graphical user interface for a quick entry of the values of the necessary patient variables, followed by the prediction of the need for EUS. The user-friendly interface (Video core tip, currently not deployed on Web) provides the user with the model-based estimated probability of the patient stratification to microlithiasis/sludge and other-pancreatitis. Moreover, it provides several graphical presentations to illustrate the impact of the specific variables on the decision. A multicenter prospective score validation with harmonised predefinition of biliary sludge and microlithiasis is currently being planned.

CONCLUSION

We present for the first time an ML-based tool, externally validated in two sets of data from tertiary pancreatic referral centers, to predict the presence of biliary sludge and microlithiasis in patients with an initial label of idiopathic pancreatitis with an accuracy of 0.7607 (95%CI: 0.673-0.8347), PPV of 0.7573, and NPV of 0.7857. Upon prospective validation, the prediction score will aid in decision-making on which patient to subject to EUS for diagnostic workup at a first episode of pancreatitis.

ARTICLE HIGHLIGHTS

Research background

About 30% of acute pancreatitis (AP) cases classified as idiopathic actually have a biliary and thus monocausally treatable origin.

Research motivation

To date, there is no predictive score to differentiate between idiopathic and sludge- and microlithiasis-triggered acute biliary pancreatitis. Undiagnosed biliary pancreatitis aetiology poses the risk of overdiagnosis and additional patient burden. AP triggered by small biliary concrements (microlithiasis and sludge) is a particularly challenging diagnosis.

Research objectives

The aim of this study was to develop a machine-learning based prediction score for the presence of microlithiasis and sludge in AP patients. External score validation was performed at two university pancreas centres.

Research methods

The clinical and laboratory parameters of 218 AP patients were used to calculate a machine-learning based prediction model for the presence of sludge and microlithiasis. Forty-seven patients with endosonographic evidence of sludge and microlithiasis (and no other possible underlying pancreatitis aetiology) were used in the identification cohort and compared with 171 AP patients without endosonographic evidence of sludge and microlithiasis. We trained supervised machine learning classifiers using H₂O.ai automatically selecting the best suitable predictor model to predict microlithiasis/sludge. An external pancreatitis cohort from two university pancreas centres with 117 patients was used for validation.

Research results

The score, constructed from a total of 28 simple variables to be collected in the early phase of pancreatitis-associated hospitalisation and validated externally at two university pancreas centres, can predict the presence of biliary sludge and microlithiasis with an accuracy of 0.7607 (95% confidence interval: 0.673-0.8347), positive predictive value of 0.7573, and negative predictive value of 0.7857.

Research conclusions

For the first time, we present a machine-learning based prediction score to differentiate between sludge- and microlithiasis-triggered AP and idiopathic pancreatitis. By using it in the early phase of pancreatitis-related hospitalisation, patient selection for or against the use of endosonography can support clinical decision-making.

Research perspectives

Upon prospective validation, the prediction score will aid in decision-making on which patient to subject to endosonography for diagnostic workup at a first episode of pancreatitis specifically to differentiate between sludge/microlithiasis-triggered and idiopathic AP.