Domain specific training with three-dimensional virtual colonoscopy images improves artificial intelligence based automated size prediction for large lesions

Abstract

BACKGROUND

Accurate measurement of the lesion size in colonoscopy is important for selecting the guideline-recommended resection methods and surveillance intervals. The commonly applied technique of visual size estimation by endoscopists can be inaccurate by up to 66%.

AIM

To develop, use, and compare artificial intelligence-based methods to improve size estimation.

METHODS

We used three state-of-the-art monocular depth estimation models. Furthermore, one model was fine-tuned by domain-specific training with 3060 white-light-rendered images and depth maps of a virtual colonoscopy through a three-dimensional colon model. We evaluated the models using single white-light images obtained from two independent cohorts: (1) One internal cohort of 133 patients who received an endoscopic submucosal dissection (ESD) (mean lesion diameter 31.4 mm); and (2) One external cohort of 42 patients with polyps (mean diameter 5.2 mm) receiving a snare-assisted polypectomy. The gold standard for size estimation was caliper measurement of the lesions after removal. We analyzed the absolute diameter and European Society of Gastrointestinal Endoscopy guideline conform predefined size bins as an outcome parameters. For comparison in the ESD cohort, three expert and three trainee endoscopists estimated the lesion size.

RESULTS

The mean absolute percentage error (MAPE) in the ESD cohort for the three models ranged from 68.3% to 77.0%. The fine-tuned model reached a MAPE of 37.0%. The experts and trainees achieved 27.3% and 37.5%, respectively. In the external cohort, the MAPE of the three models ranged from 37.6% to 102.4%, while the fine-tuned model achieved the best result with 33.9%.

CONCLUSION

While monocular depth estimation holds potential for automating polyp size prediction, bigger lesions pose an additional challenge and require rigorous domain-specific training using methods such as three-dimensional virtual colonoscopy.

Key Words: Polyp size; Lesion size; Colonoscopy; Artificial intelligence; Monocular depth estimation

Core Tip: Accurate measurement of colorectal polyp size is mandatory influencing the choice of resection technique and surveillance intervals. However, conventional visual estimation by endoscopists is prone to significant error rates up to 66%. In this work, we compare three state-of-the art monocular depth estimation artificial intelligence models for automated size measurement. When applied natively to colonoscopy, they show poor accuracy in size prediction. By domain specific training using virtual colonoscopy images we could significantly improve size prediction. The fine-tuned model achieved a mean absolute percentage error of 33.9%-37.0%. This performance was comparable to trainee endoscopists but remained inferior to experts.

INTRODUCTION

The detection and removal of polyps via colonoscopy substantially reduces the incidence and mortality of colorectal cancer[1]. The choice of the resection technique and the determination of post-polypectomy surveillance intervals are primarily dictated by the lesion size. The guidelines from major societies, such as the European Society of Gastrointestinal Endoscopy (ESGE) and the United States Multi-Society Task Force on Colorectal Cancer, explicitly rely on defined size thresholds (e.g., 5 mm, 10 mm) for risk stratification[2,3]. However, there is currently no easy method to predict size with high enough accuracy in real time. The current standard is the endoscopist’s ability to estimate the polyp size during the procedure. Nevertheless, studies have repeatedly demonstrated that this estimation accuracy is often intolerably low, with rates varying between 25% and 66%[4-6]. This subjective estimation is additionally prone to interobserver variability and a phenomenon known as “terminal digit preference”, where measurements cluster conveniently around 5 mm, 10 mm, and 15 mm rather than reflecting the true dimensions[7]. While physical tools such as open biopsy forceps, rulers, or snares can be used as reference standards, they often require the use of a one-way device, which can extend procedure time. Furthermore, such devices can sometimes be difficult to position in angulated segments of the colon prone to peristaltic and, in the worst case, raise the CO₂ footprint of the procedure.

Machine learning is a rapidly evolving scientific field with many applications in gastrointestinal endoscopy, such as polyp detection, characterization, and quality control[8,9]. However, literature using deep learning to precisely measure object dimensions within the endoscopic view is still sparse. Previous articles have proposed network architectures that classify polyps into broad groups (e.g., diminutive vs large)[10]. Recently, one paper has successfully used artificial intelligence (AI)-based monocular depth estimation for poly size estimation[11]. The principal advantage of this approach is that polyp size estimation can be derived directly from the image, eliminating the need for specialized endoscopes or additional physical instruments.

In this study, we used and compared deep learning-based systems based on monocular estimation to predict colorectal polyp size from single white-light endoscopy images. We used or fine-tuned three state-of-the-art monocular estimation models to generate depth maps and validated the system’s ability to predict absolute lesion diameter in two collectives, namely an internal cohort of lesions with a large diameter and an external cohort with small-diameter polyps. We analyzed the absolute diameter as an outcome parameter. Furthermore, we evaluated the size prediction in the clinical relevant ESGE guideline conform predefined size bins (≤ 5 mm, 6-9 mm, 10-19 mm, 20-39 mm, and > 39 mm), where choice of the recommended resection may method change. For the internal cohort, we additionally compared the AI result with the estimations of three experts and three trainees. The gold standard for comparison was lesion measurement with a ruler or caliper after removal.

MATERIALS AND METHODS

Three-dimensional virtual colonoscopy

For generating training data for the depth estimation model, we used a three-dimensional (3D) colon model[12] and simulated multiple colonoscopies in Blender (Version 3.0, Blender Studio, Netherlands), an animation platform. In total, 3060 sequential white-light–rendered images were obtained. For each image, a 16-bit depth map was further generated to represent the depth from the virtual camera to the 3D surface for each pixel.

Monocular depth estimation models

For generating the depth maps, three depth estimation models were compared: (1) Apple™ depth pro[13]; (2) UniK3D[14]; and (3) Depth Anything V2[15]. All of these models were trained on indoor or outdoor scenes by their developers. Moreover, we fine-tuned the Depth Anything V2 model with the 3060 virtual colonoscopy images for domain-specific adaptation by splitting the data into two parts, 80% for training an 20% for testing. For size estimation, the biggest diameter in the white-light image was taken and measured in pixels. The center of the diameter was taken as the depth reference point. Each model evaluated the distance from the camera to this point. Following this process, the pinhole camera approximation was used to calculate the size of the lesions. To assess differences between the models and the real endoscopy images, a common factor was calculated for each model by randomly sampling and least squares fitting 10% of the real sizes to the estimated sizes.

Statistical analysis

Two cohorts were used for the model validation. The first cohort consisted of 133 Lesions from patients who had undergone endoscopic submucosal dissection (ESD) at our hospital between 2021 and 2025. We retrospectively collected representative white-light endoscopy images taken before the ESD procedure with Olympus endoscopes (CF-EZ1500 and CF-HQ190 colonoscopes; Olympus Medical Corporation, Tokyo, Japan). After being pinned to a cork plate alongside a ruler for ground truth reference, each lesion was photographed. For validation of the models, a depth map was generated for each image. The size estimation of the models was calculated as described above. We report the mean absolute percentage error (MAPE) as the main outcome parameter. Furthermore, each lesion size was estimated by three expert endoscopists (> 2000 colonoscopies) and three trainees (< 100 colonoscopies).

The second cohort consisted of 42 videos of 42 polyps from the publicly available external Polyp-Size dataset[16]. From the videos, 42 representative white-light images were extracted. The authors measured the polyps after removal with vernier calipers. The depth map for each of the images was generated with the three models and the fine-tuned model. The estimated polyp size of the models was calculated as discussed in the paragraph above. The MAPE was taken as the main outcome parameter.

RESULTS

Absolute size prediction

Lesion size distributions for both validation cohorts are summarized in Figure 1 and Table 1. The ESD cohort exhibited a mean lesion size of 31.4 mm (range: 7.1-73.3 mm), whereas the external dataset showed a mean polyp size of 5.2 mm (range: 2.8-11.7 mm).

Figure 1 Violin plot displaying ground truth lesion size distribution of the endoscopic submucosal dissection and the external dataset. Average lesion size was much smaller in the external dataset compared to the endoscopic submucosal dissection set. ESD: Endoscopic submucosal dissection.

Table 1 Results from the two validation cohorts, mean (minimum-maximum).

Dataset	Size (mm)	Mean absolute percentage deviation (%)
	Gr.Trth	AplDepth Pro	UniK3D	DphAnyV2	DphAnyV2 3DVCol	Expert	Trainee
Endoscopic submucosal dissection dataset	31.4 (7.1-73.3)	68.3	75.3	77.0	36.0	27.3	37.5
External dataset	5.2 (2.8-11.7)	102.4	55.0	37.6	33.9

The first column displays the mean ground truth size, with the minimum and maximum sizes. The second to fifth rows show the mean absolute percentage errors from the three models, Apple Depth Pro, UniK3D, and Depth Anything V2 with no fine-tuning, as well as the fine-tuned Depth Anything V2 model (DphAnyV2_3DVCol). The last two rows show the expert and trainee results for the endoscopic submucosal dissection dataset.

The evaluation of the finetuned Depth Anything V2 model on the 20% holdout virtual colonoscopy test set demonstrated high accuracy. Threshold accuracy δ₁ reached 0.992, with an absolute relative error of 0.045, root mean squared error of 0.18, mean absolute error of 0.097, and a scaleinvariant logarithmic error of 0.073. Figure 2A presents the image rendered from the 3D colon model alongside a representative depth map prediction from the validation set.

Figure 2 Visual inspection indicates that all models generate plausible representations of the depth structure. A: The first row shows an image from the virtual colonoscopy alongside the predicted depth map of the fine-tuned Depth Anything V2 model. Depth is encoded in grayscale, with black indicating greater distance and white indicating closer proximity to the camera; B: The second row presents a representative whitelight endoscopic image from the endoscopic submucosal dissection cohort, accompanied by depth map predictions generated by Apple Depth Pro, UniK3D, Depth Anything V2 (nonfinetuned), as well as Depth Anything V2 (fine-tuned). Qualitatively, all models provide a reasonable representation of the depth structure in the image; C: The third row illustrates a representative whitelight endoscopic image from the external dataset, together with the corresponding depth map predictions from Apple Depth Pro, UniK3D, Depth Anything V2 (non-fine-tuned), and Depth Anything V2 (fine-tuned).

Table 1 provides the MAPE across cohorts. Figure 2B presents a representative white-light image from the ESD cohort alongside the corresponding depth maps generated by three pretrained models and the fine-tuned model within the ESD dataset. Figure 2C depicts a representative white-light image from the external cohort, together with the depth maps produced by the three pretrained (non-fine-tuned) models and the fine-tuned model.

In the ESD cohort (Figure 3A), MAPEs were 68.3% for Apple Depth Pro, 75.3% for UniK3D, and 77.0% for Depth Anything V2 before finetuning. Finetuning with the virtual colonoscopy images reduced the deviation of the optimized model (DepthAnyV2_CstBlrd) to 36.0%, comparable to that of the endoscopist trainees (37.5%). Expert endoscopists achieved the lowest MAPE (27.3%). Real vs predicted polyp sizes for the optimized model and experts are shown in Figure 2B, with a Bland-Altman analysis presented in Figure 2C.

Figure 3 Most prediction errors lie within the limits of agreement, suggesting that the model’s measurements are largely consistent with the gold-standard caliper measurements. A: In the endoscopic submucosal dissection cohort, the first plot illustrates the mean absolute percentage error across the three baseline models and the finetuned model. The best AI model, DphAnyV2_3DVCol, showed comparable mean percentage deviation as the trainees, while the experts still did more accurate predictions. The second plot depicts the prediction of the fine-tuned Depth Anything V2 model (DphAnyV2_3DVCol) and the experts prediction compared with the ground-truth size. Model and Experts show a similar error distribution. The third plot presents the Bland-Altman analysis for the best fine-tuned Depth Anything V2 model. The majority of this models predictions lie within the limits of agreement, suggesting good agreement of with the gold-standard caliper measurements; B: Results from the external dataset. The first graph shows the mean absolute percentage errors for the three baseline models and the fine-tuned model. The second plot depicts the prediction of the finetuned Depth Anything V2 model against the ground-truth size. Smaller polyp sizes lead to less outliers than larger sizes. The third plot presents the Bland-Altman analysis for the best fine-tuned Depth Anything V2 model.

Performance in the external cohort is presented in Figure 3B and Table 1. MAPEs were 102.4% for Apple Depth Pro, 55.0% for UniK3D, and 37.6% for nonfinetuned Depth Anything V2. Fine-tuning further improved accuracy, achieving the lowest MAPE (33.9%). Figure 2B and C display the real vs predicted polyp sizes plot and Bland-Altman plot.

Binned size prediction

Next, the accuracy of size prediction was analyzed by determining the proportion of measurements correctly classified into the clinical relevant, ESGE guideline compliant, predefined size bins (≤ 5 mm, 6-9 mm, 10-19 mm, 20-39 mm, and > 39 mm) across the two independent datasets.

Table 2 shows the overall accuracy of the size predictions averaging over all bins. In the ESD dataset, the AI model DphAnyV2_3DVCol showed the second best prediction value with 59.8% only almost as good as the experts with 61.9%. The trainees raters were again worse with 39.9%.

Table 2 Results from the two validation cohorts.

Dataset	Mean correctly classified binned Sizes (%)
	AplDepth Pro	UniK3D	DphAnyV2	DphAnyV2 3DVCol	Expert	Trainee
Endoscopic submucosal dissection dataset	53.5	50.3	37.0	59.8	61.9	39.9
External dataset	42.9	52.4	54.8	59.5

Displayed is the overall correctly classified binned sizes, i.e. if the models or humans have correctly predicted the polyp size within the bins: (1) 0-5 mm; (2) 6-9 mm; (3) 10-19 mm; (4) 20-39 mm; and (5) > 39 mm.

Evaluating the results with respect to the bins, for the ESD dataset (Figure 4A), among all predictors, expert raters demonstrated the most consistent classification accuracy across size categories, ranging from 49% in 10-19mm to 68% in 20-39mm bin. Trainee performance was comparably high for smaller measurements (10-19mm: 64%) but declined substantially for larger sizes, reaching only 10% in > 39 mm bin, suggesting a systematic underestimation in this range. Among the four AI models, DphAnyV2_3DVCol has the highest accuracy in the larger size bins (20-39 mm: 68%), performing comparably to expert raters. The non domain specific DphAnyV2 consistently returned the lowest classification accuracy across all populated bins (10-19 mm 28%; 20-39 mm: 41%; > 39 mm: 41%).

Figure 4 Percentage of correctly binned size predictions per size category for four artificial intelligence models (AplDepthPro, UniK3D, DphAnyV2, DphAnyV2_3DVCol). A: Percentage of correctly binned size predictions per size category for four artificial intelligence (AI) models (AplDphPro, UniK3D, DphAnyV2, DphAnyV2_3DVCol), pooled experts, and pooled trainees. Measurements were classified into five bins (≤ 5 mm, 6-9 mm, 10-19 mm, 20-39 mm, > 39 mm). Across the three well-populated bins, experts showed the most consistent accuracy (49%-68%), while trainees performed well for smaller sizes (Bin 3: 64%) but markedly declined for larger measurements (Bin 5: 10%). Among AI models, DphAnyV2_3DVCol achieved the highest accuracy in the larger size ranges; B: Percentage of correctly binned size predictions per size category for four AI models (AplDepthPro, UniK3D, DphAnyV2, DphAnyV2_3DVCol) in a dataset restricted to smaller measurements. All models performed best in Bin 3, with three of four reaching 75% accuracy. Performance was notably weaker in Bin 2 across all models, with AplDepthPro achieving the lowest accuracy (17%).

In the external polyp dataset (Figure 4B), measurements were confined to the smaller size range, with cases distributed across 0 mm to 19 mm. All four AI models achieved their highest accuracy in the 10-19 mm bin, with AplDepthPro, DphAnyV2, and DphAnyV2_3DVCol each correctly classifying 75% of cases, and UniK3D reaching 50%, Classification accuracy was markedly reduced across all models in the 6-9 mm bin, with values ranging from 17% (AplDepthPro) to 42% (UniK3D).

DISCUSSION

In this study, we presented and compared novel AI-based systems employing monocular depth estimation to determine colorectal polyp size from white-light endoscopic images alone. Outcome parameters were defined based on absolute size and ESGE guideline-compliant predefined size ranges. The domain-adapted and fine-tuned system, trained on 3D virtual colonoscopy images, demonstrated performance comparable to endoscopist trainees in absolute size assessment and superior performance in binned size ranges, although expert endoscopists continued to outperform the system across both metrics. The results of this categorical analysis were comparable to those obtained using absolute size measurements. Expert endoscopists maintained the highest predictive accuracy, although their performance was very close to that of the best-performing AI models. Notably, our analysis revealed that trainees, in particular, exhibited difficulty accurately predicting the diameters of larger lesions. Our tool could be deployed as an automated arbiter to assess polyp size. Determining the correct polyp lesion size in colonoscopy remains a challenge. Currently, most endoscopists estimate the polyp size during the procedure. Studies have consistently demonstrated that the estimation accuracy varies between 25% and 66%[4-6]. Accurate size estimation is mandatory for choosing the appropriate resection method and the surveillance interval. Chaptini et al[17] found that incorrect visual estimation could lead to incorrect choice of the surveillance interval in up to 10% of the cases. Previous attempts to standardize polyp estimation during colonoscopy have mainly proposed physical devices, such as snare-based rulers or biopsy forceps. Technical solutions have also been proposed. Fujifilm developed an endoscope that can project a laser at a known angle to the camera to estimate the distance to the lesion, accompanied by a projected virtual scale. van Bokhorst et al[18] used this system and found a significantly reduced variance in polyp site measurements of both experts and trainees. Some papers have also explored deep learning methods to extract absolute metric dimensions from endoscopic images. Itoh et al[19] developed a deep learning model to classify polyps into three size categories: (1) 1-5 mm; (2) 6-9 mm; and (3) 10-35 mm. Their model achieved an accuracy of 74% on the test dataset. Unfortunately, these classification models provide only a rough estimate rather than a precise measurement. Sudarevic et al[4] developed an AI-based method using a water jet as a measurement reference. They reported a median percentage system error of 7.7% when using the AI-based method vs 22.1% when using visual estimation only. However, their method still needs an additional manual step, enabling and aiming the water jet, to determine the polyp size. Our approach using monocular depth estimation potentially provides an easy-to-implement method for existing commercially available AI systems for polyp detection and characterization, fully automatizing size estimation. Wang et al[11] developed and used a similar monocular depth estimation model to ours, trained by virtual colonoscopies. They validated their system with 157 polyps with a mean size of 6.4 mm, ranging from 4.0 mm to 8.0 mm. Their system reached a MAPE of 16.3%, which was significantly better compared to the visual size estimation of expert endoscopists (> 3000 colonoscopies) with a MAPE of 32.3%. In our results, we found a similar deviation among the experts’ estimates. Our system, however, performed worse than the experts in the internal ESD cohort. Wang et al[11] used DenseNet-169, which was proposed for monocular depth estimation by Alhashim and Wonka[20], as the backbone in their work. One of the standard datasets for performance comparison of monocular depth estimation methods is NYU Depth v2[21]. The DenseNet-169 backbone reached an absolute relative error of 0.103 on this set. This result is significantly worse than state-of-the-art models like the one used in this paper, Depth Anything V2, which reached an absolute relative error of 0.044 (-57.4%, compared to 0.103) on the same dataset. Surprisingly, Wang et al[11] still obtained excellent results with DenseNet-169. The authors reported that they used multiple 3D colon models for training, while we used only one. We speculate that using a variety of 3D models with different diameters and geometries may improve our model. Future investigations are needed to confirm this hypothesis. In our study, we evaluated the models on two cohorts with significantly different sizes. The mean lesion size in the internal ESD cohort was six times higher than that of the external cohort. We observed that the even among the natively trained models, with one exception (Apple Depth Pro), the depth estimation results performed better in the external cohort with smaller lesions. We speculate that the size of the lesions poses an additional challenge for automated measurement methods because of these methods’ 3D structure and the increased distance to the lesion from the camera. To our knowledge, our study is the first to comparatively evaluate multiple state-of-the-art monocular depth estimation models within the context of colonoscopy. Our findings highlight a critical limitation in current computer vision applications for endoscopy: General-purpose models – typically trained on indoor, outdoor, or autonomous driving datasets – do not generalize well to the complex geometry of the colon. Moreover, they perform even worse with larger lesions. When applied natively, these models demonstrated a high average MAPE of 73.5%, confirming a significant lack of generalizability. However, we found that this limitation can be effectively mitigated through transfer learning using synthetic data from virtual colonoscopies. By fine-tuning the Depth Anything V2 model with these domain-specific data, the error rate was substantially reduced to 37.5%. While this study relied on 3D virtual imaging, we speculate that future accuracy could be further enhanced by incorporating real endoscopic videos paired with high-precision ground truth data, such as LiDAR (light detection and ranging) measurements. The present study has several limitations, including the specific hardware used for validation. The system was validated primarily using endoscopes from one brand, Olympus, although of different generations. Different focal lengths and lens characteristics across manufacturers could influence the generalizability of the depth estimation method. Second, we used only single images to determine the polyp size, and no videos were used. The stability of the approach needs to be validated in real time; we plan a future study to address this issue. Third, in our study, we determined the lesion size after removal. In our internal ESD cohort, spanning the lesion across the cork plate may have altered the size because of the flexibility of the tissue. light detection and ranging. In the external cohort, sucking the polyp through the working channel could have also altered its geometry, leading to inaccurate ground truth.

CONCLUSION

In conclusion, while monocular depth estimation holds potential for automating polyp size prediction. However, larger lesions pose an additional challenge to such systems. Successful clinical implementation requires rigorous domain adaptation. Transfer learning with 3D virtual colonoscopy images provides a scalable, robust method to improve the accuracy of these systems, bringing them closer to the performance levels of human operators.