Predicting lymph node metastasis in colorectal cancer using case-level multiple instance learning

Abstract

BACKGROUND

The accurate prediction of lymph node metastasis (LNM) is crucial for managing locally advanced (T3/T4) colorectal cancer (CRC). However, both traditional histopathology and standard slide-level deep learning often fail to capture the sparse and diagnostically critical features of metastatic potential.

AIM

To develop and validate a case-level multiple-instance learning (MIL) framework mimicking a pathologist's comprehensive review and improve T3/T4 CRC LNM prediction.

METHODS

The whole-slide images of 130 patients with T3/T4 CRC were retrospectively collected. A case-level MIL framework utilising the CONCH v1.5 and UNI2-h deep learning models was trained on features from all haematoxylin and eosin-stained primary tumour slides for each patient. These pathological features were subsequently integrated with clinical data, and model performance was evaluated using the area under the curve (AUC).

RESULTS

The case-level framework demonstrated superior LNM prediction over slide-level training, with the CONCH v1.5 model achieving a mean AUC (± SD) of 0.899 ± 0.033 vs 0.814 ± 0.083, respectively. Integrating pathology features with clinical data further enhanced performance, yielding a top model with a mean AUC of 0.904 ± 0.047, in sharp contrast to a clinical-only model (mean AUC 0.584 ± 0.084). Crucially, a pathologist’s review confirmed that the model-identified high-attention regions correspond to known high-risk histopathological features.

CONCLUSION

A case-level MIL framework provides a superior approach for predicting LNM in advanced CRC. This method shows promise for risk stratification and therapy decisions, requiring further validation.

Key Words: Colorectal cancer; Lymph node metastasis; Deep learning; Multiple instance learning; Histopathology

Core Tip: To better predict lymph node metastasis (LNM) in advanced colorectal cancer, this pilot study developed a case-level deep learning framework. By analysing the pathology slides of all patients and emulating a pathologist's workflow, the model achieved a high area under the curve of 0.899, outperforming traditional methods. Integrating the clinical data further increased the accuracy to 0.904. This interpretable approach is a promising tool for refining LNM risk assessments and guiding adjuvant therapy decisions.

INTRODUCTION

Colorectal cancer (CRC) is the leading cause of cancer-related mortality worldwide[1]. Among its stages, T3 and T4 CRC represent locally advanced disease, characterised by tumour invasion through the muscularis propria into the perirectal or pericolic fat (T3) or further into adjacent structures (T4)[2]. A defining factor in managing these stages is the presence of lymph node metastasis (LNM), which critically determines the prognosis and necessity for adjuvant chemotherapy after surgical resection[3,4]. Compared with T1 and T2 tumours confined to the bowel wall and often node-negative, rarely warranting adjuvant therapy, T3 and T4 tumours, with a higher propensity for LNM, frequently do so, especially when lymph nodes are involved (Stage III)[5]. However, even in node-negative T3 or T4 cases, the risk of occult metastases or recurrence persists and complicates treatment decisions. The current standard histopathological assessment of resection samples is limited by sampling inconsistencies and the potential to miss micrometastases, leading to inaccurate LNM classification and suboptimal therapeutic strategies[6,7].

Deep-learning models have significantly advanced pathological image analysis and classification. Techniques such as convolutional neural networks (CNNs), transformers, and ensemble networks have proven effective in classifying pathological images and enhancing the accuracy, consistency, and efficiency of medical diagnostics, particularly for tumour detection and grading[8-10]. Deep learning (DL) has revolutionised computational pathology, extending its applications beyond diagnosis to uncover novel pathological evidence, including predicting biomarkers, molecular alterations, and LNM from histological data, as well as enabling virtual staining[11-14]. However, their effectiveness in these tasks remains limited, highlighting the potential for further development.

Recent advances in DL have demonstrated its potential for predicting LNM in CRC using histopathological images. Although CNNs combined with clinical data achieve moderate performance, with an area under the curve (AUC) of approximately 0.74, their accuracy is still comparable to, and not significantly better than, traditional histopathological assessments[15,16]. This limitation may stem from methodological constraints: Current models typically rely on slide-level labels during training, which assume a uniform distribution of high-risk features (e.g. lymphovascular invasion, perineural invasion, and tumour budding) across all slides[17]. However, these features are often sparsely distributed, requiring pathologists to meticulously review entire slide sets for accurate diagnosis. Slide-level training may miss these critical but infrequent indicators, potentially limiting the generalisation of the model. To address this issue, we propose a case-level multiple-instance learning (MIL) framework that mirrors the comprehensive evaluation performed by pathologists. By aggregating information across all slides per patient, this approach aims to improve the detection of high-risk features, enhance the robustness of LNM prediction, and uncover clinically relevant pathological patterns for T3/T4 CRC.

MATERIALS AND METHODS

Data collection

A cohort of 153 colorectal adenocarcinoma cases diagnosed between 2023 and 2024 was retrospectively identified from the archives of Chongqing Traditional Chinese Medicine Hospital. The inclusion criteria were pathological stage T3 or T4 primary tumours, a minimum of 12 lymph nodes examined in the resection specimen, availability of at least four primary tumour slides per case, and no history of neoadjuvant therapy[18]. After applying these criteria, 130 patients met the eligibility criteria and were included in the final study cohort (Figure 1). This study was approved by the Institutional Review Board of Chongqing Traditional Chinese Medicine Hospital. For each included case, all available haematoxylin and eosin (H&E)-stained slides derived from primary tumour resection were used, and slides containing metastatic lymph node tissue were excluded from this analysis. The ground truth for LNM status (positive or negative) was established based on a definitive histopathological assessment of all resected lymph nodes, as documented in the final surgical pathology report. The selected primary tumour slides were subsequently digitised using an F.Q. CytoSense 40P scanner (Guangzhou F.Q. PATHOTECH Co., Ltd.) at 20 × objective magnification (0.8 Numerical Aperture, 0.1760 µm/pixel resolution).

Figure 1 Diagram of inclusion/exclusion criteria for colorectal cancer data cohorts. LN: Lymph nodes.

Data processing

Whole-slide images (WSIs) captured at 20 × magnification were segmented into non-overlapping tiles using QuPath. Tumour areas within the WSIs were annotated independently by two pathologists (Wang CL and Zou LF) to guide tile selection. Tiles containing more than 50% background were excluded, and background pixels were identified using the Otsu thresholding method to determine the optimal brightness threshold. The Reinhard method was applied to normalise the tile colours. To ensure robust dataset splitting and prevent data leakage, patient-level 5-fold Monte Carlo cross-validation was employed to randomly partition the cases into training and validation sets across five iterations. This approach ensures that all slides and tiles from a single patient belong exclusively to either the training or the validation set within any given fold, thus preventing the model from being tested on data from patients that it has already seen during training. All the models in this study utilised the same resulting split datasets for consistency in training and validation.

DL strategy and feature extraction

In this study, a clustering-constrained attention MIL method[19] was applied to perform instance-level clustering of histopathological images without manual annotation. The hybrid neural network architecture illustrated in Figure 2 integrates feature extraction and clustering to analyse WSIs effectively. Each histopathological image was pre-processed at a resolution of 0.5 μm/pixel to meet the input requirements of the feature extraction models. Regions of interest (ROIs) were resized to 256 × 256 pixels for UNI2-h and 512 × 512 pixels for CONCH v1.5, followed by normalisation using ImageNet parameters [mean: (0.485, 0.456, 0.406); SD: (0.229, 0.224, 0.225)][20,21]. Features were extracted using UNI2-h by loading the model weights from a designated repository (https://huggingface.co/MahmoodLab/UNI2-h). The model processed 256 × 256 ROIs in inference mode, generating feature embeddings with a dimensionality of 1536 per ROI. Features were concurrently extracted using CONCH v1.5 by loading model weights from a designated repository (https://huggingface.co/MahmoodLab/conchv1_5). The model processed 512 × 512 ROIs in inference mode, producing embeddings with a dimensionality of 768 per ROI. Feature extraction was performed in an NVIDIA GPU-enabled environment, and the resulting features were stored as NumPy arrays in HDF5 format. These embeddings serve as inputs for the clustering-constrained attention MIL framework to enable downstream instance-level analysis.

Figure 2 Workflow of case-level multiple instance learning for lymph node metastasis prediction in colorectal cancer histopathology. A: Data acquisition and preprocessing. Hematoxylin and eosin (H&E) stained slides from primary tumor resections are scanned using a digital scanner to generate whole-slide images (WSIs). WSIs are then processed into smaller, non-overlapping patches for subsequent analysis; B: Feature extraction with slide-level and case-level labeling. Slide-level label: For each slide, patches are extracted and fed into the feature extractor. Case-level label: Patches from all slides belonging to a single patient case are processed by the feature extractor; C: Multiple instance learning (MIL) Framework for lymph node metastasis (LNM) Prediction and Interpretation. MIL: Feature embeddings from patches of a case are input into the clustering-constrained-attention MIL framework. Attention scoring is applied to assign importance weights to different patches. Clustering is used to group similar patches. Pooling mechanisms aggregate these attention scores to generate a case-level or slide-level prediction. Attention scoring is applied to assign importance weights to different patches, potentially highlighting diagnostically relevant regions. Pooling mechanisms aggregate these attention scores to generate a case-level prediction. LNM prediction based on deep learning (DL): The MIL framework, utilizing deep learning, outputs a prediction for LNM (negative or positive). Integration: The DL based LNM prediction is integrated with clinical data to potentially enhance prediction accuracy. Machine learning: Integrated clinical and pathology features can be further analyzed using traditional machine learning classifiers to generate a final LNM prediction. H&E: Hematoxylin and eosin; WSIs: Whole-slide images; MIL: Multiple instance learning; LNM: Lymph node metastasis; ML: Machine learning; DL: Deep learning.

Combination of feature data

Feature data from multiple HDF5 files, each containing patch coordinates and embeddings from histopathological images, were combined into a single HDF5 file. To prevent overlap, the X-coordinates of the patches from subsequent files were shifted by an offset of 768 units relative to the maximum X-coordinates of the prior data. The coordinates and features were concatenated across files, and the resulting datasets were saved in a new HDF5 file with the attributes copied from the first file.

Determination of the optimal number of clusters

The elbow method was employed to identify the optimal number of clusters (denoted as k). The hierarchical clustering dendrogram was iteratively cut using the cutree function to obtain cluster assignments for k values ranging from 1 to 30. For each value of k, the sum of squared errors (SSE) was calculated in the Uniform Manifold Approximation and Projection (UMAP) space as follows. For each cluster, the centroid was determined as the mean of the UMAP coordinates of all points assigned to that cluster, and the SSE was computed by summing the squared Euclidean distances from each point to its respective cluster centroid. This process quantified the within-cluster variability for each k.

Dimensionality reduction using UMAP

To analyse and visualise the high-dimensional feature data extracted from the histopathological images, UMAP, a nonlinear dimensionality reduction technique, was employed. UMAP was applied to the principal component scores that had been derived from the preceding principal component analysis (PCA). The dimensionality of the data was reduced using PCA and the most significant variance was preserved. This step facilitated the exploration of complex patterns within the data in a computationally efficient and interpretable manner.

Histopathological interpretation of model-selected patches in training data

To further investigate the histopathological features that drive the predictive capacity of the model, a targeted review of the high-attention regions was conducted using only the training dataset. True-positive cases within the training set were identified and defined as cases with confirmed LNM that were accurately predicted as positive by the model. From these true positive cases, the slide exhibiting the highest predicted probability of LNM was selected; specifically, slides exceeding a probability threshold of 0.9 were prioritized. Subsequently, these selected slides were reprocessed using the trained DL model to extract the image patches that corresponded to the highest attention weights within the model architecture. These high-attention patches, representing the regions most influential in the positive predictions of the model during training, were then subjected to detailed histopathological interpretation by two experienced pathologists (Wang CL and Zou LF). Pathologists, blinded to the model’s output, independently reviewed the patches to identify the predominant histological patterns. Any discrepancies were resolved by a consensus discussion to arrive at a final interpretation for each identified cluster.

Machine learning classifiers

To develop models for LNM prediction, a diverse set of machine-learning algorithms were employed, each with a distinct architectural foundation. Ensemble methods, such as random forest and extremely randomised trees, were included, where predictions were aggregated from numerous decision trees to achieve robust results. Gradient boosting frameworks, including XGBoost and LightGBM, were also utilised in which refined models were built sequentially by correcting errors from the previous stages. Support vector machines (SVM), architecturally designed to define optimal separating boundaries in high-dimensional space, were also employed. Furthermore, a Multilayer Perceptron, a neural network architecture capable of learning complex patterns through interconnected layers of nodes, was incorporated. These varied architectures, implemented using the validated scikit-learn library in Python, provided a comprehensive approach for predictive modelling in this study.

Model interpretability and feature contribution analysis

To provide interpretability for the machine-learning models, a post-hoc analysis was conducted using SHapley Additive exPlanations (SHAP). A model-agnostic kernel-based SHAP methodology was employed, which is appropriate for explaining nontree-based architectures, such as SVMs. SHAP values were subsequently calculated for each patient in the corresponding validation set. These values quantified the marginal contribution of each input feature, including the DL-derived pathology score and all clinical variables, which were transformed via one-hot encoding, to the model's predicted probability for the LNM-positive class. The results were visualised via SHAP summary plots to evaluate both the overall feature importance, ranked by the mean absolute SHAP value, and the directional influence of the feature values on the model output.

Computational environment

The DL models were developed in Python (version 3.11.5) using the PyTorch framework (version 2.3.1+cu118) running on an Ubuntu-based workstation with an Intel Core i9-14900KF CPU, 128 GB of RAM, and an NVIDIA GeForce RTX 4090 GPU.

Evaluation of the DL-based model

A comprehensive assessment was conducted to evaluate the predictive performance of each machine learning model for LNM. The AUC was utilised as the primary metric to quantify the discriminative ability of each model, reflecting its capacity to distinguish between cases with and without LNM. In addition to the AUC, a range of other performance metrics were calculated to provide a multifaceted evaluation, including accuracy, sensitivity, specificity, positive predictive value, negative predictive value, precision, recall, and F1-score. These metrics were employed to assess various aspects of model performance, such as the overall correctness, ability to correctly identify positive cases (sensitivity), ability to correctly identify negative cases (specificity), and balance between precision and recall (F1-score).

RESULTS

Baseline characteristics

This study analysed a cohort of 130 patients with CRC (59 LNM-positive and 71 LNM-negative), from whom 1016 primary tumor H&E slides were digitised for analysis. Baseline demographic and clinical characteristics of the patients are shown in Table 1. Multivariate logistic regression (LR) was subsequently performed to identify the independent clinical predictors of LNM. This analysis revealed that only patients in the 70-79 age group were independently associated with lower odds of LNM (odds ratio 0.24, 95%CI: 0.09-0.63, P = 0.004), whereas other factors like T stage, tumour location, and carcinoembryonic antigen (CEA) level were not significant (all P > 0.05; Supplementary Table 1).

Table 1 Baseline characteristics of the colorectal cancer patients included in the combined training and validation datasets (n = 130).

Characteristic		Number	Col%	n (LNM)	Row%	OR (95%CI)	P value
Sex	Male	71	55%	32	45%	1.00 (reference)	N/A
	Female	59	45%	27	46%	1.25 (0.62-2.50)	0.530
Age, years	< 60	29	22%	19	66%	1.00 (reference)	N/A
	60-69	32	25%	14	44%	0.60 (0.21-1.68)	0.327
	70-79	52	40%	18	35%	0.23 (0.09-0.62)	0.003
	> 80	17	13%	8	47%	0.37 (0.11-1.26)	0.112
CRC location	Colon	74	57%	27	37%	1.00 (reference)	N/A
	Rectum	56	43%	32	37%	1.79 (0.89-3.61)	0.104
Sideness	Proximal	39	30%	16	41%	1.00 (reference)	N/A
	Distal	91	70%	43	47%	1.11 (0.52-2.36)	0.788
CEA (ng/mL)	< 5	101	78%	45	45%	1.00 (reference)	N/A
	> 5	29	22%	14	48%	1.16 (0.51-2.66)	0.723
T stage	T3	95	73%	42	44%	1.00 (reference)	N/A
	T4	35	27%	17	49%	1.19 (0.55-2.59)	0.658

Col% represents the percentage distribution within the total cohort. n is the count of patients with lymph node metastasis (LNM) in each subgroup. Row% indicates the prevalence of LNM within each subgroup. Odds ratios (OR) and 95%CI reflect the association between each characteristic and LNM status, relative to the reference category. P-values assess the statistical significance of the ORs. CRC: Colorectal cancer; LNM: Lymph node metastasis; OR: Odds ratio; CEA: Carcinoembryonic antigen; N/A: Not applicable.

DL model performance for LNM prediction

The performance of our DL models for LNM prediction was evaluated across three key factors: Granularity of training labels (case-level vs slide-level), model architecture (CONCH v1.5, vs UNI2-h), and use of pathologist-annotated tumour regions (ROIs). All performance metrics are listed in Table 2 and Supplementary Table 2.

Table 2 Area under the receiver operating characteristic curve for lymph node metastasis prediction across models using 5-fold cross-validation.

Category	Fold 1	Fold 2	Fold 3	Fold 4	Fold 5	mean ± SD
CONCH	0.732 (0.503-0.921)	0.767 (0.550-0.945)	0.865 (0.700-0.976)	0.934 (0.825-1.000)	0.770 (0.551-0.925)	0.814 ± 0.083
CONCH c	0.903 (0.758-0.994)	0.870 (0.703-0.994)	0.893 (0.743-0.988)	0.954 (0.855-1.000)	0.877 (0.731-0.982)	0.899 ± 0.033
CONCH ROI	0.709 (0.482-0.915)	0.798 (0.598-0.944)	0.843 (0.639-0.979)	0.924 (0.787-1.000)	0.755 (0.536-0.927)	0.806 ± 0.083
CONCH ROI-c	0.759 (0.544-0.929)	0.826 (0.642-0.958)	0.884 (0.725-0.993)	0.910 (0.765-1.000)	0.809 (0.621-0.963)	0.838 ± 0.060
UNI2	0.682 (0.450-0.893)	0.775 (0.577-0.941)	0.839 (0.648-0.970)	0.830 (0.650-0.970)	0.693 (0.476-0.879)	0.764 ± 0.074
UNI2 c	0.737 (0.519-0.911)	0.607 (0.364-0.824)	0.954 (0.851-1.000)	0.775 (0.556-0.944)	0.695 (0.458-0.875)	0.754 ± 0.128
UNI2 ROI	0.699 (0.475-0.905)	0.756 (0.539-0.929)	0.844 (0.654-0.976)	0.846 (0.673-0.976)	0.726 (0.516-0.911)	0.774 ± 0.068
UNI2 ROI-c	0.750 (0.536-0.929)	0.737 (0.521-0.911)	0.892 (0.737-1.000)	0.867 (0.703-0.987)	0.819 (0.643-0.952)	0.813 ± 0.069

Values represent area under the curve scores with 95%CI in brackets for each fold, alongside the mean ± SD across folds. Models evaluated include: CONCH (CONCH v1.5 trained on slide-level labels), CONCH-c (CONCH v1.5 trained on case-level labels), CONCH-ROI (CONCH v1.5 trained on slide-level labels with tumor area annotations), CONCH-ROI-c (CONCH v1.5 trained on case-level labels with tumor area annotations), UNI2 (UNI2-h trained on slide-level labels), UNI2-c (UNI2-h trained on case-level labels), UNI2-ROI (UNI2-h trained on slide-level labels with tumor area annotations), and UNI2-ROI-c (UNI2-h trained on case-level labels with tumor area annotations). ROI: Regions of interest.

Our primary finding was that the case-level training strategy, which aggregated information from slides of all patients, consistently and significantly outperformed the slide-level approach. The CONCH v1.5 architecture achieved the highest predictive accuracy, yielding a mean AUC of 0.899 with case-level labels. This represents a marked improvement over the AUC of 0.814 achieved with the slide-level labels. The CONCH v1.5 model also proved superior to the UNI2-h architecture under identical training conditions, outperforming it at both the case-level (mean AUC 0.899 vs 0.754) and slide-level (mean AUC 0.814 vs 0.764) configurations.

Counterintuitively, explicitly guiding the models with ROI annotations did not improve, and in some cases, hindered performance. For the top-performing CONCH v1.5 model, adding ROIs reduced the case-level AUC from 0.899 to 0.838 and had a negligible effect at the slide level (mean AUC, 0.806). Interestingly, this effect was model-dependent, as the UNI2-h model showed a modest performance benefit from ROI annotation in both the case- (mean AUC 0.754-0.813) and slide-level (mean AUC 0.764-0.774) settings.

Beyond predictive accuracy, we also assessed computational efficiency by measuring the mean epoch duration during training (Supplementary Table 3). The case-level approach demonstrated consistently superior efficiency compared with its slide-level counterpart across all tested configurations. This advantage was particularly pronounced for the CONCH v1.5 model, where case-level training significantly reduced the epoch duration, both with and without ROI annotations (Figure 3). These results suggest that the proposed case-level MIL framework offers dual advantages in terms of predictive power and computational resource utilisation.

Figure 3 Comparison of computational efficiency across different model configurations. A: CONCH v1.5 model with pathologist-annotated regions of interest (ROI); B: CONCH v1.5 model without ROI annotations; C: UNI2-h model with ROI annotations; D: UNI2-h model without ROI annotations. Each panel displays the mean epoch duration (in seconds), comparing slide-level (black bars) and case-level (gray bars) training strategies. Data are presented as mean ± SD. Statistical significance was determined using a two-tailed unpaired t-test (A, C, D) or a Mann-Whitney U test (B). ^bP < 0.01, ^cP < 0.001, ^dP < 0.0001. ROI: Regions of interest.

Integration of clinical and pathology data for LNM prediction

Building on the findings that pathology-based DL models, particularly CONCH v1.5, with case-level labels, demonstrated robust performance in LNM prediction (achieving an average AUC of 0.899, as detailed in Table 2), we next sought to evaluate whether integrating clinical features could further enhance predictive capabilities. Building upon the finding that pathology-based DL models demonstrate robust performance, we next sought to evaluate whether integrating clinical features could further enhance predictive capabilities. To quantify this, we compared the performance of 11 machine learning classifiers trained on two feature sets: Clinical data alone vs a combination of clinical and pathological features (Supplementary Table 4). The average results from the 5-fold cross-validation are summarised in Figure 4. As illustrated by the clear separation between the ROC curves in each panel, the models trained exclusively on clinical data (red curves) exhibited limited predictive power. mean AUCs (± SD) ranged from 0.497 ± 0.044 (ExtraTrees) to 0.584 ± 0.084 (SVM). Contrastingly, integrating deep-learning-derived pathology features (blue curves) led to a dramatic and consistent improvement across all classifiers. This combined approach yielded mean AUCs ranging from 0.743 ± 0.061 (K-nearest neighbours) to a robust 0.904 ± 0.047 (SVM). This makes the SVM the top-performing method in this study. Other models also showed strong performance with the combined data, notably LR and ExtraTrees, which achieved mean AUCs of 0.889 ± 0.066 and 0.875 ± 0.06, respectively.

Figure 4 Comparative performance of machine learning models for predicting lymph node metastasis. This figure presents eleven panels, each dedicated to a different machine learning classifier. Within each panel, two receiver operating characteristic (ROC) curves are displayed to compare predictive performance based on different feature sets. The red curve represents the model trained using only clinical features ('Cli'), while the blue curve represents the model trained on combined clinical and deep learning-derived pathology features ('Cli + Pat'). The solid lines depict the mean ROC curve averaged across a 5-fold cross-validation, with the shaded areas representing the standard deviation. The corresponding mean area under the curve ± SD values for each feature set are annotated within each panel. SVM: Support vector machine; LR: Logistic regression; AUC: Area under the curve; KNN: K-nearest neighbours; GBM: Gradient boosting machine; MLP: Multilayer perceptron.

To elucidate the predictive drivers of our top-performing SVM model (mean AUC, 0.904), we employed SHAP to assess feature contributions across the five cross-validation folds (Figure 5). The analysis consistently revealed that the pathology score generated by the CONCH v1.5 model trained with case-level labels, was by far the most influential predictor of LNM status. As demonstrated in SHAP summary plots, higher pathology scores were strongly associated with an increased likelihood of positive LNM prediction. However, the remaining clinical variables, including age, T-stage, sex, CEA level, and tumour location exerted a comparatively minor influence on the model output. This interpretability analysis confirmed that the model's robust predictive power was primarily derived from the rich histopathological information captured by the case-level DL framework.

Figure 5 SHapley Additive exPlanations analysis of the top-performing support vector machine model. This figure presents a SHapley Additive exPlanations (SHAP) analysis of the top-performing support vector machine model across the five cross-validation folds. For each fold, overall feature importance is ranked by mean absolute SHAP value (bar charts, left), while corresponding summary plots (right) visualize the distribution and directional impact of SHAP values for individual predictions. In these plots, color indicates the original feature value (high in red, low in blue), revealing how feature levels drive model output. SHAP: SHapley Additive exPlanations; SVM: Support vector machine; CEA: Carcinoembryonic antigen; CRC: Colorectal cancer.

Histopathological interpretation of high-attention morphological features

To interpret the morphological basis of the model predictions, the high-attention image patches automatically identified by the model were first subjected to dimensionality reduction and unsupervised clustering. This computational analysis aims to group patches based on their learned features without prior human input. Using the elbow method, we identified an optimal number of six clusters, a finding corroborated by UMAP visualisation that demonstrated a clear separation of the feature embeddings into six distinct groups (Figure 6). This result indicated that our case-level MIL model successfully learned to stratify histopathological patterns into cohesive morphological categories, confirming its ability to identify recurring tissue phenotypes that are predictive of LNM.

Figure 6 Clustering of high-attention histopathological features to identify morphological patterns. A: Elbow plot for determining the optimal number of clusters. The sum of squared errors is plotted against the number of clusters (k). The inflection point ("elbow") and the indicator line at k = 6 suggest that six is the optimal number of clusters for this analysis; B: Uniform Manifold Approximation and Projection visualization of high-attention tile embeddings, demonstrating separation into six distinct clusters. Each point represents an individual image tile, and its color denotes assignment to one of the six clusters as defined in the legend. SSE: Sum of Squared Errors; UMAP: Uniform Manifold Approximation and Projection.

Following this computational grouping, a detailed histopathological review of representative tiles from each of the six machine-generated clusters was performed by two expert pathologists (Wang CL and Zou LF) to assign a clinical interpretation to each group (Figure 7). Their analysis revealed that each cluster corresponded to a well-established high-risk feature associated with aggressive tumour behaviour. The phenotypes identified included poorly differentiated adenocarcinoma (Cluster 1), prominent desmoplastic reaction (Cluster 2), complex glandular architecture (Cluster 3), micropapillary adenocarcinoma (Cluster 4), overt lymphovascular and perineural invasion (Cluster 5), and signet-ring cell carcinoma (Cluster 6). The autonomous identification of this spectrum of high-risk features, subsequently validated by pathologists, demonstrates that the model’s decision-making process aligns with established pathological principles, functioning not as a "black box", but as an interpretable tool for recognizing significant indicators of metastatic potential.

Figure 7 Histopathological interpretation of high-attention morphological clusters. Representative image tiles from the six distinct clusters identified by the deep learning model. Each cluster corresponds to a specific histopathological phenotype associated with lymph node metastasis risk. Cluster 1: Poorly differentiated adenocarcinoma, characterized by solid sheets and trabecular growth patterns with a high nuclear-to-cytoplasmic ratio. Cluster 2: Prominent desmoplastic reaction, showing a dense fibroblastic stromal response to invading tumor cells. Cluster 3: Adenocarcinoma with complex glandular architecture, featuring cribriform, fused, or back-to-back glands indicative of high-grade tumor organization. Cluster 4: Micropapillary adenocarcinoma, a high-risk pattern defined by small, cohesive tufts of tumor cells floating in stromal spaces without a true fibrovascular core. Cluster 5: Overt invasion, showcasing clear evidence of lymphovascular invasion and perineural invasion, where tumor cells infiltrate lymphatic channels and surround nerve fibers. Cluster 6: Signet-ring cell carcinoma, a distinct subtype composed of tumor cells containing large intracytoplasmic mucin vacuoles that displace the nucleus to the periphery.

DISCUSSION

In this study, we developed and validated a case-level MIL framework using WSIs to predict LNM in patients with locally advanced CRC. Our results demonstrate that this case-level approach, which aggregates information from all tumour slides for a given patient, significantly outperforms standard slide-level methods, achieving a mean AUC of 0.899, compared to 0.814 for slide-level training. Integrating these pathological features with clinical data further improved accuracy, with the top-performing model yielding an AUC of 0.904, in sharp contrast to a model using clinical data alone (AUC 0.584). Crucially, the clinical relevance of the model was underscored by pathological validation, which confirmed that high-attention regions correspond to known high-risk histological features predictive of LNM.

The performance of our case-level model (AUC = 0.899) demonstrated a substantial advancement in LNM prediction. It not only compares favourably to previous DL studies that achieved AUCs of approximately 0.74[15,16] but also vastly outperforms a model built exclusively from our own clinical data (AUC 0.584). Notably, the poor performance of this clinical model persisted even though our multivariate analysis identified age as a statistically significant independent predictor of LNM. This finding strongly illustrates that isolated clinical and demographic variables are insufficient for robust risk stratification. The superiority of our framework stems from its case-level MIL strategy, which emulates the comprehensive review of the entire case by a pathologist. This holistic approach is uniquely capable of capturing heterogeneous tumour characteristics and sparsely distributed prognostic features, which are the true drivers of metastatic potential.

Considering the pilot nature of this study and the inherent limitations of working with a relatively small cohort of 130 CRC cases, we employed the clustering-constrained attention MIL framework. This choice was particularly well suited for our pilot investigation because this MIL approach demonstrates notable advantages when applied to smaller datasets, which are common in exploratory medical imaging research[19]. Clustering-constrained attention MIL excels in such scenarios by enabling effective instance-level learning from bag-level labels (case level in our study), allowing the model to extract meaningful patterns from individual histopathology tiles despite the limited number of cases. Integrated clustering mechanisms enhance the organisation of the feature space, potentially improving model robustness and generalisation, even with a smaller training set typical of pilot studies. Concurrently, attention mechanisms ensure that the model prioritises the most diagnostically significant regions within each WSI for accurate LNM prediction.

The observed superiority of case-level labelled datasets over slide-level labelled datasets in our study directly reflects the inherent nature of the histopathological assessment of LNM risk. Critical diagnostic features indicative of metastasis, such as lymphovascular invasion, perineural invasion, and tumour budding, are often sparsely distributed across WSIs within a patient case. These high-risk features are not always present in every slide and can be missed if the models are trained solely on slide-level labels, which implicitly assume feature homogeneity across all slides. In clinical practice, pathologists comprehensively review all available slides for a given patient to identify these potentially infrequent yet highly prognostic features. This holistic case-centric evaluation is crucial for accurate LNM risk stratification. Our finding that case-level training significantly enhanced the model performance (AUC 0.899 vs 0.814) strongly suggests that this approach more effectively captures the subtle but critical information distributed across the entire patient case, thereby closely mirroring and potentially augmenting the diagnostic acumen of expert pathologists. Although a direct comparison with pathologists was not performed, the model that achieved this level of accuracy demonstrated a high level of diagnostic accuracy for this task. This performance significantly surpasses that of models that rely on less comprehensive data or clinical factors.

Furthermore, our findings reveal a performance advantage for the CONCH v1.5 model over UNI2-h across most training paradigms. This superiority extends beyond the predictive accuracy to include model stability and robustness. While both models were trained on identical case-level cross-validation splits, CONCH v1.5 delivered highly consistent performance (mean AUC 0.899 ± 0.033), whereas UNI2-h exhibited significant instability, with its AUC fluctuating from 0.607 to 0.954 between folds (mean AUC 0.754 ± 0.128). This disparity may be attributed to several factors, including differences in the model architecture and input resolution. CONCH v1.5, processing larger 512 × 512 pixel patches than UNI2-h's 256 × 256, likely benefits from a broader contextual view of the histopathological landscape within each tile. In pathology image analysis, context is paramount; larger tiles can encompass more complex tissue architectures, tumour-stroma interactions, and subtle spatial relationships between different cell types, all of which could be crucial for discerning prognostic features related to LNM[22-24]. Contrastingly, the smaller ROIs processed by UNI2-h might capture finer details but potentially at the cost of losing broader contextual information.

Interestingly, manual tumour annotation did not improve - and, in some instances, even reduced - predictive performance. This counterintuitive result strongly suggests that restricting the view of the model to neoplastic cells detrimentally constrains learning capacity. We hypothesised that this is because the accuracy of the model depends on its ability to assess the entire tumor microenvironment (TME), a complex landscape containing competing biological signals that collectively determine metastatic potential[25,26]. The TME harbours a pro-tumourigenic stromal response. This is driven by cancer-associated fibroblasts that mediate the desmoplastic reaction, a feature that our model correctly identified in high-attention regions (Cluster 2), and actively remodels the extracellular matrix to create pathways for tumour cell dissemination[27,28]. Conversely, the TME can also mount a powerful antitumorigenic immune response organised within Tertiary Lymphoid Structures (TLS). These immune hubs are associated with a lower risk of metastasis and improved patient survival, functioning as local sites for adaptive immunity[29,30]. Both the dense, reactive stroma and lymphocytic infiltrates of TLS were visually prominent features on standard H&E slides. Therefore, it is highly plausible that a model with a holistic, unconstrained view will learn to assess the net prognostic impact of these competing forces. By forcing the model to focus only on tumour cells, we prevented it from learning this crucial biological balance. Our clustering-constrained attention MIL framework, which is designed to discover salient regions without relying on precise localisation, is uniquely suited to learning from this complete biological context, rendering explicit ROI annotations redundant and counterproductive.

A key strength of our case-level MIL framework is its profound interpretability. The attention mechanism of the model effectively mirrors a pathologist's diagnostic process by focusing on high-risk features, a finding quantitatively confirmed through unsupervised clustering of high-attention regions. This analysis demonstrated that the model did not operate as an uninterpretable "black box" but instead learned to autonomously identify and group a validated spectrum of histopathological risk factors directly associated with metastatic potential in T3/T4 CRC. Our subsequent pathological review of the machine-generated clusters sequentially and systematically validated their clinical significance.

The model first identified Cluster 1, which was characterised by poorly differentiated adenocarcinoma. This finding is highly significant because loss of glandular differentiation is a fundamental hallmark of aggressive tumour biology and a potent predictor of metastatic disease[31,32]. By prioritising these regions, the model correctly learned to associate high-grade cytological atypia and disorganised growth patterns with an increased LNM risk. Next, the model focuses on Cluster 2, which features a prominent desmoplastic reaction. This highlights its capacity to recognise the dynamic tumour-stroma interface, where a dense fibroblastic response signifies an active, aggressive invasion process integral to tumor progression[33,34].

The analysis then identified high-risk architectural patterns. Cluster 3 included regions defined by complex glandular structures, including cribriform and fused glands. Although these patterns are known prognostic factors in other cancers, in colorectal adenocarcinoma their significance lies in their contribution to tumour grading. Such complex and disorganised structures are hallmarks of high-grade morphology, signifying a loss of normal glandular architecture. This high-grade status is a powerful and well-established predictor of aggressive tumour behaviour, including increased invasiveness and a higher propensity for LNM[35-40]. Therefore, the specific attention of the model to these features demonstrates its ability to identify a critical component of the tumour grading system used by pathologists to assess metastatic risk. Following this, the model identified Cluster 4, which consisted of micropapillary adenocarcinomas. The specific identification of this pattern is particularly noteworthy, as pathologists recognise this variant as a well-established high-risk subtype with a strong independent association with LNM and adverse outcomes[41-43].

Crucially, the model proved adept at providing the most direct evidence of metastatic capability. Cluster 5 contained unequivocal examples of overt lymphovascular and perineural invasions. The ability to detect these focal events, which represent the primary conduits for tumour dissemination, is a testament to the model's clinical utility and its potential to augment a pathologist's review by flagging these critical high-yield regions[44,45]. Finally, Cluster 6 isolated another aggressive subtype, signet-ring cell carcinoma. The model's specific attention to this variant is highly relevant given that it is notorious for its rapid clinical course and high propensity for lymphatic spread[46-51]. Collectively, the sequential identification of these six distinct high-risk phenotypes validates our model's alignment with established pathological principles and underscores its power as a robust and interpretable predictive tool.

This study has several strengths. First, the introduction of a case-level MIL framework directly addresses the critical methodological gap in DL-based histopathological analysis for LNM prediction. By moving beyond slide-level labelling and mirroring the comprehensive, multi-slide evaluation inherent to pathological practice, we demonstrated a tangible improvement in predictive performance. The robust AUC achieved, particularly with the CONCH v1.5 model and further enhanced by integrating clinical data, underscores the clinical potential of this approach. Moreover, the qualitative validation provided by an expert pathologist review of the high-attention tiles lends significant biological plausibility to our model, demonstrating its ability to identify and prioritise features that align with the established pathological knowledge of CRC metastasis. However, as a pilot study, there are inherent limitations that must be acknowledged. First, the retrospective nature of our study and its reliance on data from a single institution inherently limits the generalisability of our findings to broader patient populations and diverse clinical settings. The relatively small sample size of 130 patients, which was sufficient to establish a proof-of-concept for our case-level approach, was modest for DL applications. Therefore, validation in larger multicentre cohorts is a crucial next step to validate the robustness and clinical utility of our model. Future research should also prioritise the inclusion of greater demographic and geographic diversity to ensure that the model performs equally across different patient groups. Furthermore, our focus on T3/T4 tumours, while clinically relevant for adjuvant therapy decisions, restricted the applicability of our conclusions to specific stages of CRC. Future research should explore the performance of this case-level MIL framework across the entire spectrum of CRC stages and datasets encompassing patients who received neoadjuvant therapy. Finally, while the pathologist review provided valuable insights, a more quantitative approach to characterise and validate the features within the high-attention tiles could further strengthen the interpretability and clinical translatability of our findings.

This study provides a promising avenue for improving LNM risk assessment in patients with locally advanced CRC. Our case-level MIL framework demonstrates a potential improvement in predictive accuracy compared with slide-level methods, which could contribute to more informed adjuvant therapy decisions in the future. The integration of clinical data further enhances the practicality of this approach in the existing clinical workflows. Moreover, this study adds to the growing body of evidence supporting the utility of MIL in computational pathology for histopathological analyses. The model's identification of pathologically relevant features, as validated by an expert review, provides a basis for further exploration of DL methodologies for feature discovery in cancer. Future research should build on these findings to quantitatively investigate the identified histological features and explore the broader applicability of case-level MIL strategies in other diagnostic areas.

CONCLUSION

Our study demonstrates that a case-level DL approach improves the prediction of LNM in advanced CRC. This method, which analyzes the entirety of patient pathology slides, better reflects clinical practice and enhances accuracy compared with traditional slide-based analysis. The combination of image-based features with clinical data further strengthens this prediction. Importantly, an expert pathologist review validated that the model identified clinically relevant tumour characteristics. These results indicate that case-level DL holds significant promise for refining LNM risk assessments in CRC and advancing computational pathology.