Deep-learning-based object detection of pelvic autonomic nerves during total mesorectal excision

Abstract

BACKGROUND

Recognition of pelvic autonomic nerves (PAN) during total mesorectal excision (TME) largely depends on the surgeon’s expertise, making it susceptible to misrecognition and unintentional damage. There is an urgent need for objective and real-time support methods.

AIM

To develop a deep learning (DL) model for precise recognition and visual annotation of 5 categories of PAN during TME.

METHODS

This single-center retrospective study enrolled 120 TME videos from January 2021 to January 2023. A total of 3246 high-quality images were obtained and split 9:1 into training and internal test sets. Difficult-to-recognize characteristics were summarized. An additional 20 independent TME videos from June 2023 to January 2024 were used for external validation. The DL model performance was compared with that of surgeons and verified pathologically. χ², Fisher’s exact test and t-tests were applied (P < 0.05).

RESULTS

The DL model achieved a precision of 0.839, a recall of 0.769, and a mean average precision at intersection over union 50 of 0.873. The overall recognition rate in external validation was 76.0%, similar to senior surgeons (73.9%, P = 0.156) but superior to junior surgeons (64.9%, P = 0.001). The miss rate of 5 PAN categories ranged from 12.9% to 29.6%. Initial recognition time (2.08-2.21 seconds) was shorter than that of senior surgeons (5.65-6.19 seconds, P < 0.01); mean continuous tracking duration was prolonged by 57.21-66.45 seconds compared with that of senior surgeons (P < 0.01). Low nerve exposure caused most DL model false negatives, while cord-like fibrous tissue dominated false positives. All 7 harvested specimens were pathologically confirmed to contain nerve tissue, with a processing speed of 25 frames per second.

CONCLUSION

The model demonstrates recognition performance comparable to that of senior surgeons, with pathological confirmation. It may potentially help preserve PAN during TME and shorten the learning curve for junior surgeons.

Keywords: Rectal cancer; Total mesorectal excision; Pelvic autonomic nerve; Deep learning; Object detection

Core Tip: In this study, a deep learning model was developed to recognize five categories of pelvic autonomic nerves during total mesorectal excision. The model was trained, validated and tested on surgical video images, and its performance was compared with surgeons at different levels. The model achieved precision and speed comparable to senior surgeons, with consistent pathological confirmation of all 7 sampled nerve specimens. Ultimately, this model was confirmed to be reliable and may assist nerve preservation and shorten the learning curve for junior surgeons.