Deep-learning-based object detection of pelvic autonomic nerves during total mesorectal excision

Abstract

BACKGROUND

Recognition of pelvic autonomic nerves (PAN) during total mesorectal excision (TME) largely depends on the surgeon’s expertise, making it susceptible to misrecognition and unintentional damage. There is an urgent need for objective and real-time support methods.

AIM

To develop a deep learning (DL) model for precise recognition and visual annotation of 5 categories of PAN during TME.

METHODS

This single-center retrospective study enrolled 120 TME videos from January 2021 to January 2023. A total of 3246 high-quality images were obtained and split 9:1 into training and internal test sets. Difficult-to-recognize characteristics were summarized. An additional 20 independent TME videos from June 2023 to January 2024 were used for external validation. The DL model performance was compared with that of surgeons and verified pathologically. χ², Fisher’s exact test and t-tests were applied (P < 0.05).

RESULTS

The DL model achieved a precision of 0.839, a recall of 0.769, and a mean average precision at intersection over union 50 of 0.873. The overall recognition rate in external validation was 76.0%, similar to senior surgeons (73.9%, P = 0.156) but superior to junior surgeons (64.9%, P = 0.001). The miss rate of 5 PAN categories ranged from 12.9% to 29.6%. Initial recognition time (2.08-2.21 seconds) was shorter than that of senior surgeons (5.65-6.19 seconds, P < 0.01); mean continuous tracking duration was prolonged by 57.21-66.45 seconds compared with that of senior surgeons (P < 0.01). Low nerve exposure caused most DL model false negatives, while cord-like fibrous tissue dominated false positives. All 7 harvested specimens were pathologically confirmed to contain nerve tissue, with a processing speed of 25 frames per second.

CONCLUSION

The model demonstrates recognition performance comparable to that of senior surgeons, with pathological confirmation. It may potentially help preserve PAN during TME and shorten the learning curve for junior surgeons.

Key Words: Rectal cancer; Total mesorectal excision; Pelvic autonomic nerve; Deep learning; Object detection

Core Tip: In this study, a deep learning model was developed to recognize five categories of pelvic autonomic nerves during total mesorectal excision. The model was trained, validated and tested on surgical video images, and its performance was compared with surgeons at different levels. The model achieved precision and speed comparable to senior surgeons, with consistent pathological confirmation of all 7 sampled nerve specimens. Ultimately, this model was confirmed to be reliable and may assist nerve preservation and shorten the learning curve for junior surgeons.

INTRODUCTION

Total mesorectal excision (TME) combined with autonomic nerve preservation (ANP) is the standard surgical procedure for radical resection of rectal cancer, which can preserve sexual and urinary functions on the basis of radical tumor resection[1,2]. The key pelvic autonomic nerves (PAN) requiring intraoperative protection mainly include the inferior mesenteric plexus (IMP), superior hypogastric plexus (SHP), hypogastric nerves (HN), pelvic splanchnic nerves (PSN), inferior hypogastric plexus (IHP) also known as pelvic plexus (PP), neurovascular bundle (NVB), and nerves innervating the levator ani muscle[3,4]. Although three dimensional laparoscopic and robotic surgical systems have enabled more precise visual recognition of PAN and reduced the incidence of postoperative genitourinary dysfunction compared with open surgery, approximately one-third of patients still suffer from functional impairment due to unintentional intraoperative PAN injury. Specifically, the incidence rates of urinary and sexual dysfunction are 20%-45% and 2.7%-100% after three dimensional laparoscopic surgery, and 20%-30% and 0%-74% after robotic surgery, respectively[5-8]. Notably, there is still no universally accepted standardized protocol for PAN preservation in clinical practice. In addition, PAN injury during TME is not classified as a surgical complication in major mainstream guidelines, and no effective salvage or therapeutic strategies are available once PAN injury occurs. These facts further highlight the clinical urgency of precise intraoperative recognition and protection of PAN.

In recent years, artificial intelligence (AI) technology, especially deep learning (DL), has achieved significant breakthroughs in medical image segmentation, object detection and other fields, driving the development of intelligent precision surgery[9,10]. At present, AI has been widely applied to instrument recognition, surgical task tracking and objective evaluation of surgical skills in laparoscopic surgery[11-13]. In robot assisted rectal cancer surgery, AI has also been demonstrated to enable real time recognition of anatomical structures including fascial layers, resection planes, colon, inferior mesenteric artery, and seminal vesicle, which can effectively reduce visual interference in the narrow surgical space and help surgeons avoid tissue injury[14,15]. Several studies have explored optimized strategies for intraoperative PAN preservation. Some studies focused on preoperative individualized magnetic resonance neuroimaging to delineate the course and variation of PAN; however, small nerve branches and anatomical variations remain difficult to visualize clearly[16,17]. Intraoperative PAN monitoring mainly includes electrophysiological monitoring and AI monitoring. The former is relatively complex in workflow and carries potential risks of tissue trauma, whereas AI monitoring exhibits the advantages of simple operation and easy popularization, better aligning with clinical practical needs. Nevertheless, current AI models are limited by several drawbacks, including incomplete coverage of PAN categories, idealized training data, lack of pathological verification, and absence of comparison with surgeons of different seniority. Studies on real time recognition of the full spectrum of PAN are scarce, hindering clinical translation.

The You Only Look Once (YOLO) v10 model has achieved dual breakthroughs in both speed and precision for small-object detection, boasting the advantages of fewer parameters and excellent real-time performance[18]. It provides crucial technical support for the rapid recognition of slender PAN intraoperatively, yet has not been utilized in studies on the concurrent recognition of various categories of PAN. Based on the above clinical demands and research gaps, this study aimed to develop and validate a real-time multi-category PAN recognition model using YOLOv10. Data processing was refined in combination with the surgical field of view, and the DL model performance was rigorously evaluated through multi-dimensional validation to provide evidence for its clinical translation, thereby reducing the incidence of PAN injury and improving patients’ postoperative quality of life.

MATERIALS AND METHODS

Study design

This was a retrospective diagnostic study that enrolled TME surgery videos from two periods. A total of 120 TME surgeries performed in the Department of Colorectal Surgery, Fujian Medical University Union Hospital from January 2021 to January 2023 were used for DL model training and internal testing, while 20 surgeries performed at the same center from June 2023 to January 2024 served as an independent external validation cohort. The inclusion criteria were age ≥ 18 years, confirmed pathological diagnosis, surgical videos with clear visualization of PAN, and laparoscopic or robotic surgical procedures, while low-quality surgical videos with low resolution, blurred images, severe bleeding, or other conditions impairing annotation quality were excluded (Figure 1).

Figure 1 Flowchart of the study design. PAN: Pelvic autonomic nerves; YOLO: You Only Look Once; DL: Deep learning.

Image acquisition and preprocessing

Video clip screening: Junior surgeons with clinical experience < 10 years extracted video clips containing PAN from 120 TME videos, and two senior surgeons with clinical experience ≥ 10 years and an annual surgical volume ≥ 600 cases classified the video clips into three groups according to PAN recognition confidence: > 90% indicating clear morphology without severe tissue adhesion or bleeding interference, 50% indicating blurred boundaries with moderate interference, and < 10% indicating indistinct structures with severe interference; any discrepancies were resolved by a third senior surgeon with 20 years of experience, and 210 video clips with confidence > 90% were finally selected for subsequent annotation and DL model training. For the external validation cohort, 20 TME videos were clipped into 54 PAN-containing video clips by junior surgeons, and after evaluation by two operating senior surgeons, 13 video clips with severe interference or misrecognition were excluded, leaving 39 video clips for analysis.

Image acquisition: All surgical videos were in two dimensional MP4 format with a resolution of 1920 × 1080 pixels and a frame rate of 30 frames per second, with all patient identifiers removed. Initial frames were extracted at a sampling rate of 1 frame per second. After redundancy removal (similarity threshold = 0.58) and quality control using Python and OpenCV-including exclusion of blurry, overexposed, or occluded frames (> 50% occlusion), as well as frames without PAN regions. Ultimately, 3246 high-quality images were obtained from 210 training and internal test video clips, saved in JPG format and named as “video clip ID-image number” for annotation, dataset partitioning, and traceability.

Image annotation and consistency assessment

The PP, also known as the IHP, is a sympathetic and parasympathetic neural network composed of the PSN and HN; because intraoperative distinction between the PSN and PP is challenging, both structures were uniformly labeled as pelvic nerve (PN). Annotation was performed independently by two senior surgeons with more than 10 years of clinical experience, and inter-annotator agreement (IAA) was assessed using the Dice coefficient, defined as 2 × |A∩B|/(|A| + |B|), where A and B represent the annotation regions of the two surgeons, with values ranging from 0 to 1 (1 indicating complete overlap and 0 indicating no overlap)[19]; if the IAA was < 0.5, a third senior surgeon with 20 years of clinical experience made the final decision to confirm the PAN annotation boxes.

Model architecture and training

Dataset partitioning and training strategy: The 3246 images were divided into a training set (2900 images) and an internal test set (346 images) at a 9:1 ratio. To avoid model overfitting and improve model’s recognition robustness and generalization ability, a combined data augmentation strategy was applied to the training set images, including clockwise rotation at various angles, horizontal/vertical/central flipping, image translation by 5%-10% of the side length, and brightness adjustment of ± 10%-20% to simulate intraoperative interference. All transformations were applied synchronously to the images and the corresponding neural annotations. The internal test set was used for baseline model performance evaluation.

Model selection and training environment: The YOLOv10 model was used in this study. This model demonstrates 1.8 × higher inference speed than real-time detection transformer-R18 and a 2.8 × reduction in parameters, showing superior real-time performance and efficiency. The DL model training environment was configured as follows: Windows 10 64-bit operating system, NVIDIA Tesla P40 graphics processing unit (GPU), Python 3.10.15 scripting language, PyTorch 2.3.0 DL framework, and CUDA 11.8 with the cuDNN 8.9.5 GPU acceleration library.

DL model performance evaluation

DL model evaluation metrics: DL model detection performance was evaluated using precision, recall, F1-score, mean average precision at intersection over union (IoU) 50 (mAP50), and mean average precision from IoU 50 to 95 (mAP50-95); intraoperatively, branches of the HN and SHP to the colorectum recognized by the DL model were marked with Hemlock clips and sent for pathological examination to verify recognition precision, and neurological complications were followed up postoperatively.

Independent external validation: Manual annotations by two operating senior surgeons served as the gold standard; The recognition rate and miss rate of the DL model were compared with those of non-operating senior and junior surgeons, the recognition performance of the DL model for each PAN category was analyzed in subgroups, and the initial recognition time and continuous tracking duration of PAN by the DL model and senior surgeons were further compared. All participating surgeons watched the complete and continuous surgical videos under unified experimental conditions without any permission to pause, replay, or slow down the video playback. All surgeons were fully informed of the corresponding surgical phases in advance, ensuring identical information acquisition conditions for both human observers and the DL model.

False-positive and false-negative recognition analysis

Of 60 video clips misjudged by junior surgeons and with confidence < 10% according to senior surgeons were screened, and false-positive recognition characteristics were summarized based on the positive outputs of the DL model for these 60 clips. Five-fold cross-validation was conducted on the training set containing 2900 images, with each fold trained 10 iterations at a confidence threshold of 0.001, and images in which PAN was not recognized in all 10 iterations were collected to summarize false-negative recognition characteristics.

Statistical analysis

Statistical analysis were performed using SPSS 26.0 and R 4.5.1 software; normally distributed data were presented as mean ± SD, non-normally distributed data as median (interquartile range), between-group comparisons were conducted using the χ² test or Fisher’s exact test, annotation consistency was evaluated using the Kappa test, and P value < 0.05 was considered statistically significant.

RESULTS

Patient baseline characteristics

The internal cohort consisted of the training set and the internal test set. In the internal cohort and the independent external cohort, the mean age of patients was 62.5 ± 10.2 years and 63.1 ± 9.8 years, with a mean body mass index of 23.1 ± 2.8 kg/m² and 22.8 ± 3.0 kg/m², respectively. Most patients received preoperative neoadjuvant therapy, accounting for 73.3% in the internal cohort and 75.0% in the independent external cohort. A total of 5402 annotation boxes were included in the internal cohort and 930 in the independent external cohort, covering nine labels of PAN: Left inferior mesenteric plexus, right inferior mesenteric plexus, SHP, left hypogastric nerve, right pelvic nerve, left neurovascular bundle and right neurovascular bundle, as shown in Table 1.

Table 1 Patients baseline characteristics and distribution of pelvic autonomic nerves annotation boxes, mean ± SD/n (%).

Characteristic	Internal cohort (n = 120)	External validation cohort (n = 20)
Gender
Male	68 (56.7)	11 (55.0)
Female	52 (43.3)	9 (45.0)
Age	62.5 ± 10.2	63.1 ± 9.8
BMI (kg/m2)	23.1 ± 2.8	22.8 ± 3.0
UICC-TNM stage
II stage	87 (72.5)	14 (70.0)
III stage	33 (27.5)	6 (30.0)
Histological type
Adenocarcinoma	97 (80.8)	18 (90.0)
Other	23 (19.2)	2 (10.0)
Surgical procedure
Low anterior resection	99 (82.5)	17 (85.0)
Abdominoperineal resection	3 (2.5)	0 (0.0)
Intersphincteric resection	18 (15.0)	3 (15.0)
Intraoperative blood loss (mL), median (interquartile range)	15 (10-20)	12 (8-18)
Operation duration (minute)	256.7 ± 71.4	256.7 ± 71.4
Preoperative neoadjuvant therapy
Received	88 (73.3)	15 (75.0)
Not received	32 (26.7)	5 (25.0)
Annotation box (sum)	5402	930
RIMP	973 (18.0)	140 (15.1)
LIMP	864 (16.0)	157 (16.9)
SHP	216 (4.0)	37 (4.0)
RHN	610 (11.3)	110 (11.8)
LHN	648 (12.0)	114 (12.3)
RPN	820 (15.2)	130 (14.0)
LPN	878 (16.2)	158 (16.9)
RNVB	200 (3.7)	44 (4.7)
LNVB	193 (3.6)	40 (4.3)

BMI: Body mass index; UICC-TNM: Union for International Cancer Control-tumor node metastasis; RIMP: Right inferior mesenteric plexus; LIMP: Left inferior mesenteric plexus; SHP: Superior hypogastric plexus; RHN: Right hypogastric nerve; LHN: Left hypogastric nerve; RPN: Right pelvic nerve; LPN: Left pelvic nerve; RNVB: Right neurovascular bundle; LNVB: Left neurovascular bundle.

Efficiency of the DL model in PAN recognition

After 650 training iterations and hyperparameter optimization, the recognition performance of the DL model was summarized in Table 2. After merging left and right PAN labels (IMP, HN, PN, NVB), the overall recognition performance of the DL model was significantly improved, with a precision of 0.839, recall of 0.769, F1-score of 0.802, mAP50 of 0.873, and mAP50-95 of 0.534. Among the five categories of PAN, NVB showed the highest F1-score (0.845), PN achieved the optimal precision (0.881), and IMP exhibited an mAP50 of 0.907, indicating favorable recognition performance for all PAN categories.

Table 2 Comparison of deep learning model performance metrics between merged labels and separate labels.

Label	Image	Annotation Box	DL model 1					DL model 2
			Precision	Recall	F1-score	mAP50	mAP50-95	Precision	Recall	F1-score	mAP50	mAP50-95
All	346	521	0.762	0.506	0.608	0.580	0.246	0.839	0.769	0.802	0.873	0.534
IMP	346	183						0.889	0.767	0.824	0.907	0.539
RIMP	346	97	0.764	0.568	0.652	0.606	0.238
LIMP	346	86	0.801	0.635	0.708	0.706	0.291
SHP	346	20	0.763	0.590	0.665	0.656	0.293	0.755	0.721	0.738	0.835	0.527
HN	346	123						0.851	0.704	0.771	0.842	0.520
RHN	346	61	0.777	0.489	0.600	0.550	0.248
LHN	346	62	0.797	0.450	0.575	0.540	0.248
PN	346	157						0.881	0.780	0.827	0.873	0.548
RPN	346	82	0.738	0.407	0.525	0.490	0.216
LPN	346	75	0.635	0.395	0.487	0.469	0.203
NVB	346	38						0.821	0.871	0.845	0.911	0.534
RNVB	346	20	0.755	0.492	0.596	0.592	0.251
LNVB	346	18	0.826	0.527	0.643

Precision = TP/(TP + FP) (TP: True positive, correctly identified autonomic nerve regions; FP: False positive, non-nerve regions misidentified as autonomic nerves). Recall = TP/(TP + FN) (FN: False negative, autonomic nerve regions not detected); F1-score = 2 × (precision × recall)/(precision + recall) (ranges from 0 to 1; a value closer to 1 indicates better overall performance). mAP50 indicate mean average precision across all categories at an intersection over union threshold of 0.5 (average precision is the area under the precision recall curve). mAP50-95 indicate mean average precision values at intersection over union thresholds from 0.5 to 0.95 (10 thresholds in 0.05 increments), evaluating detection robustness across different thresholds. Model 1: Deep learning model with separate labels; Model 2: Deep learning model with merged labels; DL: Deep learning; IMP: Inferior mesenteric plexus; RIMP: Right inferior mesenteric plexus; LIMP: Left inferior mesenteric plexus; SHP: Superior hypogastric plexus; HN: Hypogastric nerve; RHN: Right hypogastric nerve; LHN: Left hypogastric nerve; PN: Pelvic nerve; NVB: Neurovascular bundle; RPN: Right pelvic nerve; LPN: Left pelvic nerve; RNVB: Right neurovascular bundle; LNVB: Left neurovascular bundle.

Visualization of PAN recognition results by the DL model

The object detection results were presented in Figure 2. The DL model enabled precise recognition and visual annotation of the five PAN categories in single high-resolution frames during TME. In the representative frame, the recognition confidence scores were 0.9 for IMP, 0.7 for SHP, 0.8 for HN, 0.9 for PN, and 0.5 for NVB. The annotated regions of each nerve correlated well with their anatomical locations and distribution patterns. Five TME procedures were included, and 7 nerve branch specimens were successfully obtained, all of which were clearly recognized by the DL model and safely harvested. All patients recovered smoothly postoperatively without sexual or urinary dysfunction. Pathological examination confirmed nerve tissue presence in all 7 sampled specimens. (Figure 3). The processing speed of the system for high-resolution images was 25 frames per second (see Video), which enabled real-time continuous output of PAN localization labels with confidence scores and provided intuitive, intraoperative anatomical navigation for surgeons.

Figure 2 Visualization of object detection results by the deep learning model. A-E: First row: Manual annotation boxes with confidence > 90%; F-J: Second row: Corresponding prediction boxes by the deep learning model. IMP: Inferior mesenteric plexus; SHP: Superior hypogastric plexus; HN: Hypogastric nerves; PN: Pelvic nerve; NVB: Neurovascular bundle.

Figure 3 Intraoperative marking of pelvic autonomic nerves branches labeled by the deep learning model and pathological examination results. A: Hypogastric nerves (HN) predicted image and colorectal branch of HN; B: Superior hypogastric plexus (SHP) predicted image and colorectal branch of SHP; C: Nerve tissue is visible on pathological examination (S100 staining) (indicated by black arrow). IMP: Inferior mesenteric plexus; SHP: Superior hypogastric plexus; HN: Hypogastric nerves.

Comparison of recognition rate and miss rate among the DL model, junior surgeons, and senior surgeons

In the independent external validation cohort, 39 video clips containing 580 images and 930 annotation boxes were included for comparative analysis. Regarding recognition performance, the overall recognition rate of the DL model was 76.0% (707/930) with a miss rate of 24.0%, which was comparable to that of senior surgeons (recognition rate 73.9%, miss rate 26.1%, P = 0.156) but significantly superior to that of junior surgeons (recognition rate 64.9%, miss rate 35.1%, P = 0.001). Stratified subgroup analysis demonstrated that the DL model achieved the highest recognition rates (70.4%-87.1%) and the lowest miss rates (12.9%-29.6%) for each category of PAN. Compared with the DL model, senior surgeons showed no significant differences in recognition rates (67.9%-84.5%) or miss rates (15.5%-32.1%) for all PAN categories (all P > 0.05), whereas junior surgeons exhibited significantly lower recognition rates (54.1%-73.8%) and higher miss rates (26.2%-45.9%) (all P < 0.05), as shown in Table 3.

Table 3 Comparison of recognition rates and missed diagnosis rates for 5 categories of pelvic autonomic nerves among the deep learning model, junior surgeons and senior surgeons, n (%).

Category	Annotation box	Group	Recognized cases	Missed cases	P value
IMP	297	DL model	228 (76.7)	69 (23.3)
		Senior surgeon	222 (74.8)	75 (25.2)	0.682
		Junior surgeon	198 (66.7)	99 (33.3)	0.021
SHP	37	DL model	27 (72.1)	10 (27.9)
		Senior surgeon	26 (70.3)	11 (29.7)	0.895
		Junior surgeon	20 (54.1)	17 (45.9)	0.038
HN	224	DL model	157 (70.4)	67 (29.6)
		Senior surgeon	152 (67.9)	72 (32.1)	0.713
		Junior surgeon	132 (58.9)	92 (41.1)	0.017
PN	288	DL model	222 (77.1)	66 (22.9)
		Senior surgeon	216 (75.0)	72 (25.0)	0.758
		Junior surgeon	192 (66.7)	96 (33.3)	0.024
NVB	84	DL model	73 (87.1)	11 (12.9)
		Senior surgeon	71 (84.5)	13 (15.5)	0.812
		Junior surgeon	62 (73.8)	22 (26.2)	0.041
Sum	930	DL model	707 (76.0)	223 (24.0)
		Senior surgeon	687 (73.9)	243 (26.1)	0.156
		Junior surgeon	604 (64.9)	326 (35.1)	0.001

All P values were compared with the deep learning model group. DL: Deep learning; IMP: Inferior mesenteric plexus; SHP: Superior hypogastric plexus; HN: Hypogastric nerve; PN: Pelvic nerve; NVB: Neurovascular bundle.

Comparison of initial recognition time and tracking duration between the DL model and senior surgeons

In the independent external validation cohort, consistent recognition of PAN was achieved by the DL model and senior surgeons in 32 surgical video clips containing PAN: 28 video clips of IMP (212 annotation boxes), 15 video clips of SHP (28 annotation boxes), 30 video clips of HN (180 annotation boxes), 56 video clips of PN (223 annotation boxes), and 20 video clips of NVB (75 annotation boxes). As presented in Table 4, time-efficiency comparisons between the DL model and senior surgeons across the five PAN categories demonstrated that the DL model exhibited a significantly shorter initial recognition time for all nerve types, with a mean time advance of 3.68-3.98 seconds (t = 2.654-2.845, P < 0.01). In tracking performance analyses of PAN with varying sample sizes, the DL model showed a significantly prolonged tracking duration compared with senior surgeons, with a mean prolongation of 57.21-66.45 seconds (t = 2.857-3.158, P < 0.01).

Table 4 Comparison of initial recognition and duration of tracking time between the deep learning model and senior surgeon, mean ± SD.

Category	Group	Initial recognition time					Tracking duration
		Sample size (n)	Time (seconds)	Delta (95%CI)	t value	P value	Sample size (n)	Time (seconds)	Delta (95%CI)	t value	P value
IMP	DL model	32	2.08 ± 5.91	3.83 (1.03-6.64)	2.749	0.0081	28	199.41 ± 142.40	62.82 (19.94-107.88)	3.026	0.0061
	Senior surgeon	32	5.92 ± 8.59				28	136.59 ± 98.43
SHP	DL model	32	2.15 ± 6.12	3.91 (1.10-6.72)	2.801	0.0071	15	192.75 ± 137.80	59.34 (18.76-101.92)	2.914	0.0071
	Senior surgeon	32	6.06 ± 8.85				15	133.41 ± 95.90
HN	DL model	32	1.97 ± 5.63	3.68 (0.98-6.38)	2.654	0.0091	30	204.17 ± 145.60	64.93 (20.61-111.25)	3.092	0.0051
	Senior surgeon	32	5.65 ± 8.21				30	139.24 ± 100.70
PN	DL model	32	2.12 ± 6.03	3.87 (1.07-6.67)	2.783	0.0081	56	207.32 ± 148.20	66.45 (21.15-113.75)	3.158	0.0041
	Senior surgeon	32	5.99 ± 8.73				56	140.87 ± 101.90
NVB	DL model	32	2.21 ± 6.25	3.98 (1.15-6.81)	2.845	0.0061	20	188.63 ± 134.50	57.21 (17.98-98.44)	2.857	0.0081
	Senior surgeon	32	6.19 ± 9.02				20	131.42 ± 93.80

¹Compared with the senior surgeon group.

CI: Confidence interval; DL: Deep learning; IMP: Inferior mesenteric plexus; SHP: Superior hypogastric plexus; HN: Hypogastric nerve; PN: Pelvic nerve; NVB: Neurovascular bundle.

Analysis of false-positive and false-negative results of the DL model

Among 320 video clips, 60 video clips (18.75%) were misjudged as containing PAN by junior surgeons (with a confidence level < 10% according to senior surgeons), whereas the DL model only recognized 25 false-positive clips (7.81%). The classification of false-positive tissues in the two groups was listed in Table 5. The most common false-positive characteristics in the junior surgeon group were cord-like fibrous tissue (30%), vascular-related structures (23.33%), fat-related structures (18.33%), peritoneal incision and mesenteric tissue (15%), and surgery-related artifacts (including electrocoagulation eschar, sutures, instrument reflection, and smoke residue) (13.33%), as illustrated in Figure 4A-E. Five-fold cross-validation was performed on the 2900 images in the training set, five-fold cross-validation was conducted on the 2900 training images, and no PAN was recognized in 195 images across 10 repeated iterations. The core characteristics of false-negative images were low nerve exposure (including deep embedding in tissue, insufficient exposure, and fat adhesion) (68.7%), interference from accompanying blood vessels (14.5%), and bleeding (8.9%), while reflection, instrument traction occlusion, lens blurring or defocusing each accounted for less than 3% (Figure 4F-J).

Figure 4 Characteristics of false-positive and false-negative recognition by the deep learning model. A-E: False-positive analysis: Cord-like fibrous tissue (A); Vascular-related structures (B); Fat-related structures (C); Peritoneal incision and mesenteric tissue (D); Surgery-related artifacts (E); F-J: False-negative analysis: Deep embedding in tissue (F); Insufficient exposure (G); Fat adhesion (H); Accompanying blood vessels (I); Bleeding (J). HN: Hypogastric nerves; PN: Pelvic nerve; IMP: Inferior mesenteric plexus; NVB: Neurovascular bundle.

Table 5 False positive analysis in junior surgeon and deep learning model, n (%).

Classification of false positive recognition	Junior surgeon	DL model	Reduction of false positives
Cord-like fibrous tissue	18	9	9 (50)
Vascular-related structure	14	6	8 (57.14)
Fat-related structure	11	5	6 (54.55)
Peritoneal incision and mesenteric tissue	9	3	6 (66.67)
Surgery-related artifact	8	2	6 (75)
Sum	60	25	35 (58.33)

DL: Deep learning.

DISCUSSION

The overall visualization rate of PAN during TME remains relatively low, with 31.8% for NVB and 12.9% for PSN[20]. The narrow and deep pelvic cavity, intraoperative bleeding, and inadequate exposure of the surgical field further increase the difficulty of PAN preservation. Moreover, the recognition rate of PAN is closely associated with the surgeon’s clinical experience and anatomical knowledge reserve: For surgeons with limited clinical experience, the recognition rates of HN and SHP are 65.7% and 38.9%, respectively, while those for surgeons with extensive experience reach 81.2% and 57.6%, respectively. Most surgeons in primary hospitals and junior surgeons with less than 10 years of clinical experience lack sufficient ability to recognize PAN, which makes them more likely to cause nerve traction or inadvertent transection[20,21]. Even when surgeons intend to implement PAN preservation and PAN is visible in the laparoscopic field of view, accidental intraoperative injury to PAN still occurs in up to 34% of patients[22]. Currently, PAN recognition relies solely on the subjective judgment of surgeons, lacking objective and real-time auxiliary tools, which inevitably leads to empirical errors and hinders the standardized and homogeneous promotion of the TME combined with ANP surgical approach. AI can learn the unique characteristics of specific diseases or anatomical structures through manual annotations by medical experts[23,24]. In the field of PAN preservation, Japanese scholar Kojima et al[21] established an AI-based recognition model for HN and SHP, but the model did not cover IHP, PSN, or NVB. Domestically, Han et al[25] constructed a DL-based semantic segmentation model for multiple categories of PAN, achieving a precision of 0.749, recall of 0.659, and F1-score of 0.701, with acceptable performance confirmed by pathological examinations. Compared with semantic segmentation, object detection offers greater advantages in intraoperative PAN recognition: It does not require pixel-wise segmentation, enabling overall regional localization of nerves, and exhibits stronger anti-interference ability against complex scenarios such as defocusing, light reflection, and surgical field occlusion. Additionally, it features high computational efficiency, allowing synchronous operation with laparoscopic video streams to achieve real-time recognition and tracking, which fully meets the requirements of intraoperative real-time navigation[26]. Therefore, this study aimed to construct a non-invasive DL-based nerve recognition system based on object detection, establish a complete workflow from annotation to testing, and realize real-time recognition of all categories of PAN during TME, laying a foundation for subsequent integration into laparoscopic/robotic video systems and clinical validation.

With the advancement of minimally invasive surgical techniques, surgeons generally recognize that the anatomical structures of PAN in the IMA region and sacral promontory are relatively easy to recognize. By recognizing HN and SHP and selecting appropriate dissection planes, the preservation of deep PAN can be achieved[27,28]. However, within the bony pelvic structure, the anatomical relationships between IHP, NVB, and the surrounding fascia are relatively complex, posing significant challenges to intraoperative dissection and nerve preservation. Under endoscopic visualization, PAN is mainly manifested as pale white fibers on the fascial surface, partially accompanied by adherent blood vessels. Due to the high heterogeneity of nerves and their shallow color, it is difficult to distinguish them from surrounding fascia, tissues, and organs. Furthermore, PAN gives off multiple branches that anastomose with each other, forming a complex “neural network”. Thus, the “TME + network ANP” concept requires the preservation of not only the main trunks of PAN but also their branches and blood supply[25,29,30]. Even for experienced senior surgeons, the evaluation of network PAN is time-consuming and prone to errors. Intraoperative recognition of PAN through endoscopic lenses requires a long learning curve, which all pose considerable challenges to PAN annotation. In this study, PAN annotation was completed by two senior surgeons with ≥ 10 years of clinical experience, and the YOLOv10 model was adopted to construct the DL model. After training, the DL model achieved an overall precision of 0.839, a recall of 0.769, and an F1-score of 0.802, with an mAP50 of 0.873. Pathological verification confirmed that all 7 PAN branch specimens recognized and harvested intraoperatively by the DL model contained nerve tissue, indicating high recognition precision. Additionally, the DL model achieved full-frame recognition of continuous surgical video sequences (20-30 frames per second), and the video-level cumulative precision and recall rates reached 0.999 at 25 frames per second, which was attributed to the following factors: Surgical videos provided continuous frame sequences, and the quantitative advantage of full-frame recognition compensated for the lack of precision in single-frame detection. For example, at 25 frames per second, the cumulative precision and recall rates reached 0.999 [1 - (1-0.839)²⁵] and 0.999 [1 - (1-0.769)²⁵], respectively[25]. Moreover, intraoperative adjustment of lens parameters and tissue retraction could fully expose the network PAN, further improving the DL model’s recognition performance. In summary, the proposed DL model could effectively recognize PAN during TME + ANP surgery, providing auxiliary support for intraoperative nerve recognition.

Although quantitative indicators such as precision, recall, F1-score, and mAP50 are widely used in the evaluation of medical image object detection, these values cannot fully reflect the clinical value of the DL model for PAN recognition, especially for slender, highly variable, and low-contrast structures like PAN. In the field of computer-aided diagnosis, DL models only serve as auxiliary tools for recognition, with the final clinical decision-making authority remaining with surgeons[31]. In clinical practice, the core requirements for intraoperative nerve preservation are, on the one hand, to precisely recognize the nerves requiring protection, improve the recognition rate, and reduce the missed diagnosis rate; on the other hand, to reduce the interference from false annotations, improve the precision, and optimize the efficiency of clinical decision-making. The results of this study clearly demonstrated that the current DL model had been capable of effectively meeting the former requirement. Its recognition rate (70.4%-87.1%) was generally higher than that of senior surgeons (67.9%-84.5%), and its miss rate (12.9%-29.6%) was lower than that of senior surgeons (15.5%-32.1%). Moreover, the initial recognition time of the DL model for PAN was 3.68-3.98 seconds earlier than that of senior surgeons, and its stable tracking duration was extended by 57.21-66.45 seconds. These findings indicated that it had significant time-efficiency advantages in laparoscopic or robotic TME procedures, which could assist surgeons in recognizing PAN earlier and more stably, thereby reducing the risk of intraoperative PAN injury. Notably, the DL model and surgeons differed fundamentally in recognition patterns. The DL model implemented frame-by-frame detection, whereas surgeons judged based on continuous anatomical context and surgical experience. With consistent video viewing conditions maintained for surgeons in this study, the DL model’s speed and tracking advantages mainly reflected algorithmic traits rather than superior overall recognition performance. In addition, the application value of AI in medical education and operational evaluation has been confirmed[12]. In this study, Junior surgeons presented low PAN recognition rates and high miss rates due to limited anatomical experience. The DL model could compensate for this deficiency and had potential prospects in surgical training. Its nerve prompts might help trainees master PAN anatomy and intraoperative protection skills, and may potentially shorten the clinical learning curve.

This study had several unavoidable limitations. First, it adopted a single-center retrospective design with 140 TME videos, only including clips with a confidence level > 90% for model training (excluding interfered or unclear images). This strict selection may introduce bias, and single-center standardized equipment/techniques may limit the model’s robustness and generalization; both training and external validation datasets were from the same institution, further restricting generalization. Second, manual annotation was subjective: Blurred PAN boundaries may cause inconsistent annotation standards even among senior surgeons, and the original loose arbitration threshold may affect annotation reliability. Third, the object detection annotation-box mode, suitable for rapid intraoperative localization, failed to achieve pixel-level precise segmentation, limiting accurate display of PAN’s full course, branches, and adjacent anatomical relationships. Notably, the DL model yielded an mAP50-95 of 0.534, demonstrating inferior localization performance under stricter IoU thresholds. Bounding box deviations hindered intraoperative anatomical judgment, indicating that the model primarily served as a regional prompting tool rather than a precise anatomical navigation tool. Fourth, the study only focused on model recognition performance, neither correlating it with clinical outcomes (e.g., postoperative PAN-related complications, quality of life) nor evaluating its value in surgical education, requiring further validation via prospective studies. Additionally, independent frame-based theoretical calculations may overestimate video-level performance, and the small pathological verification sample size (only 7 nerve branch specimens) may affect result reliability. To address the aforementioned limitations, future optimization will focus on remedying deficiencies. Specific measures include: Collecting multi-center data, incorporating more images with real surgical interferences (e.g., bleeding, tissue adhesion), and iterating algorithms to enhance anti-interference ability and generalization; Integrating anatomical context (e.g., fascial layers, blood vessels) to optimize samples and reduce false positives/negatives; Adopting semi-automatic annotation and weakly supervised learning to reduce annotation cost/subjectivity and establish rigorous arbitration standards; Constructing a segment anything model-object detection hybrid framework to achieve PAN pixel-level precise segmentation[26]; Using video-level cross-validation to solve data non-independence; Expanding pathological verification sample size to improve reliability; and conducting prospective trials to validate clinical value, collecting follow-up data to confirm potential benefits in reducing PAN injury and improving postoperative quality of life and shortening the learning curve of junior surgeons, integrating the model with laparoscopic systems for real-time display, and promoting clinical translation. Furthermore, guided by emerging research on multi-omics integration and imaging-biological correlation analysis[32-34], future work will explore multimodal surgical AI with biological interpretability to break through the limitations of simple visual annotation and further strengthen the theoretical basis and translational potential of the model.

CONCLUSION

This exploratory study showed that the YOLOv10-based model could achieve real-time and precise recognition of five PAN categories during TME + ANP surgery. Its performance was comparable to that of senior surgeons and significantly superior to that of junior surgeons, with acceptable real-time performance and reliability confirmed pathologically. The DL model may provide surgeons with objective anatomical navigation. However, its potential to facilitate pelvic ANP and standardize TME + ANP procedures requires further verification. It only demonstrates preliminary translational potential in improving patient outcomes, shortening the learning curve for junior surgeons and optimizing surgical training, and more validation studies are still needed.

ACKNOWLEDGEMENTS

The authors thank all staff members of the Department of Colorectal Surgery at Fujian Medical University Union Hospital for their continued support. We also specially acknowledge the School of Medical Engineering, Henan Medical University, for providing technical support in the construction of the DL model.