Artificial intelligence-assisted endoscopic detection of esophageal neoplasia in early stage: The next step?

Abstract

Esophageal cancer (EC) is a common malignant tumor of the digestive tract and originates from the epithelium of the esophageal mucosa. It has been confirmed that early EC lesions can be cured by endoscopic therapy, and the curative effect is equivalent to that of surgical operation. Upper gastrointestinal endoscopy is still the gold standard for EC diagnosis. The accuracy of endoscopic examination results largely depends on the professional level of the examiner. Artificial intelligence (AI) has been applied in the screening of early EC and has shown advantages; notably, it is more accurate than less-experienced endoscopists. This paper reviews the application of AI in the field of endoscopic detection of early EC, including squamous cell carcinoma and adenocarcinoma, and describes the relevant progress. Although up to now most of the studies evaluating the clinical application of AI in early EC endoscopic detection are focused on still images, AI-assisted real-time detection based on live-stream video may be the next step.

Key Words: Early esophageal cancer; Artificial intelligence; Endoscopy; Diagnosis; Trend

Core Tip: Esophageal cancer (EC) is one of the most common malignant tumors of the digestive tract. Early EC lesions can be cured by endoscopic therapy, and the curative effect is equivalent to that of surgical operation. Upper gastrointestinal endoscopy is still the gold standard for EC diagnosis, while the accuracy of endoscopy depends in part on professional experience. Artificial intelligence applied in the screening of early EC has been shown to be a good assistant for those less-experienced endoscopists. This manuscript reviews the state of the art of artificial intelligence applications in clinical early EC detection by endoscopy for those who will be interested in this field.

INTRODUCTION

Esophageal cancer (EC) is one of the most common malignant tumors of the digestive tract and originates from the epithelium of the esophageal mucosa[1]. In 2018, the global incidence of EC was ranked seventh among malignant tumors (6.3/100000), and the mortality rate ranked sixth (5.5/100000)[2]. There are significant differences in the incidence and patterns of EC among different countries and regions, with the highest incidence in East Asia, which is twice the world average (12.2/100000). The main pathological type is esophageal squamous cell carcinoma (ESCC), while esophageal adenocarcinoma (EAC) is the main pathological type in relatively low-incidence areas such as Europe and America. Experiences in Japan and South Korea have confirmed that early EC lesions can be cured by endoscopic therapy. The curative effect is equivalent to that of surgical operation, and the 5-year survival rate of patients can reach 95%[3]. Therefore, early detection of EC and timely endoscopic treatment are the only ways to reduce EC mortality.

Upper gastrointestinal endoscopy is still the gold standard for EC diagnosis. After years of research and practice, the current upper gastrointestinal endoscopy diagnosis technology has made considerable progress. In terms of diagnostic methods, the original single ordinary white light imaging (WLI) has gradually developed into a variety of technologies such as pigment endoscopy, confocal laser endomicroscopy (CLE), electronic staining, magnifying endoscopy (ME), and autofluorescence imaging (AFI) endoscopy[4]. Among them, narrow band imaging (NBI) and blue laser can improve the sensitivity (SEN) of early EC diagnosis to more than 90% compared with ordinary white light endoscopy. CLE, also known as optical biopsy, can be comparable to pathological slices. The diagnostic specificity (SPE) of AFI is 50%, and the SEN is 100%[5]. At present, NBI, blue laser, etc. have been widely used in clinical practice, but clinical applications such as CLE and AFI are not universal.

British and American gastroenterologists have developed a series of guidelines for the screening and surveillance of Barrett's esophagus (BE) and EAC[6,7]. In China, gastroenterologists recommend endoscopic screening of key populations to improve the diagnosis rate of early EC. Endoscopic iodine staining and indicative biopsy screening programs implemented in high-risk areas in China can effectively reduce the incidence and mortality of esophageal cancer[8]. It is recommended that 40 years of age is the starting age for screening for EC, and screening should be terminated when the age of 75 or life expectancy is less than 5 years. Early EC and intraepithelial neoplasia (or dysplasia) are the main screening targets[9].

As we all know, the accuracy (ACC) of endoscopic examination results largely depends on the professional level of the examiner. Studies have shown that the use of proton pump inhibitors, less experienced endoscopists (< 5 years and < 1000 cases of endoscopy), and smaller lesions are significantly related to missed diagnosis of EC[10]. A great amount of needs in EC-related endoscopic screening increase the burden of clinical work, and the increase in staff fatigue may in turn affect the efficiency and ACC of the examination. How to solve the problem of the drastic increase in endoscopy workload has plagued clinical work managers.

Artificial intelligence (AI), machine learning (ML), and deep learning (DL) have relatively specific meanings, but they are usually widely used to refer to any modern processing method related to big data. AI is not "new". When Turing first designed his test, the phrase was mainly reserved for a technology that could broadly imitate human intelligence. Today, AI is widely and universally used to refer to any kind of ML program. At the most basic level, ML refers to any type of computer program that can "learn" on its own without having to be explicitly programmed by humans. ML includes supervised learning, unsupervised learning, and semi-supervised learning, which combines supervised and unsupervised approaches. In supervised learning, algorithms generate answers based on known and labeled data sets. Classification and regression algorithms, such as random forests and support vector machines (SVM), are commonly used in supervised learning. In unsupervised ML, algorithms generate answers to unknown and unlabeled data.

DL is a form of ML that can use supervised or unsupervised algorithms, or both. Through the hierarchical learning process to extract high-level and complex abstractions as data representations, DL models produce results faster than standard ML methods. Convolutional neural network (CNN) can be composed of many layers of models. Each layer takes input from the previous layer, processes it, and then outputs it to the next layer in a daisy chain.

The emergence of AI has gradually emerged in clinical work, and a large number of verifications have been obtained in medical imaging, especially the screening and diagnosis of lung cancer[11], and the analysis of pathological characteristics[12]. Some AI technologies have also been applied in the screening of early EC and have shown their advantages. This article will review the application status of AI in the screening of early EC under endoscopy and discuss its future development trends.

AI IN ENDOSCOPIC DETECTION OF ESCC

In 2002, experts of digestive endoscopy summarized the Paris classification of early digestive tract lesions[13]. Among them, the endoscopic morphology of superficial ESCCs roughly divided into three categories: Protruded, flat, and depressed types. However, in squamous epithelium, the microvascular pattern of intrapapillary capillary loops (IPCLs) is the only reliable indicator of tissue atypia[14]. In diagnostic criteria of the Japanese Esophageal Society (JES) classification, the microvascular irregularity is evaluated for the presence or absence of each of the following morphological factors: Weaving (i.e. tortuosity), dilatation, irregular caliber, and different shape (i.e. various shapes). Currently, clinical endoscopists generally use the AB classification proposed by JES. If microvessels have three or fewer factors, they are classified as type A; if they have four factors, they are classified as type B[15].

At present, NBI-ME has become the first choice clinical method for the diagnosis of early ESCC. With the help of NBI-ME, endoscopists can clearly observe the morphology of the diseased IPCL and guide the selection of targeted biopsy and clinical treatment strategies. However, the ACC of NBI-ME in diagnosing early ESCC is limited by the experience and level of endoscopists. The diagnostic ACC of experienced experts is as high as 90%, while for inexperienced or non-expert endoscopists, the diagnostic ACC is only 78.6%[16]. AI may help those inexperienced endoscopists to analyze medical images intelligently, detect and/or classify lesions, and improve the ACC of diagnosis by using variant algorithms[17] (Table 1).

Table 1 Artificial intelligence application for esophageal squamous cell cancer.

AI Application	Study design	Data category	Type of Images	AI architecture	Training dataset	Validation Method or dataset	AUC	SEN	SPE	ACC	PPV	NPV	Compared with experts	Ref.
Diagnosis	Retrospective	Still image	HRM	2-class LDA	104 sites	167 sites	0.93	84%	95%	NA	NA	NA	NA	Shin et al[18], 2016
Diagnosis	Prospective	Still image	HRM	Fully automated algorithm	104 sites	167 sites	0.937	95%	91%	NA	NA	NA	NA	Quang et al[19], 2016
Detection	Retrospective	Still image	WCE	JDPCA + CCV	400 images	10-fold-CV	0.9471	93.33%	89.20%	90.75%	NA	NA	NA	Liu et al[20], 2016
Diagnosis	Retrospective	Still image	WLI/NBI	CNN-SSD	8428 images	Caffe DL framework/1118 images	NA	81% (WLI)/89% (NBI) (per-patient) 72% (WLI)/86% (NBI) (per-image)	79%	99%	39%	95%	NA	Horie et al[21], 2019
Detection	Retrospective	Still image	WLI	DNN-CAD	2428 images	187 images	0.9637	97.8%	85.4%	91.4%	86.4%	97.6%	Superior	Cai et al[22], 2019
Diagnosis	Retrospective	Still image	WLI/NBI/BLI	CNN-SSD	22562 images	Caffe DL framework/727 images	NA	100% (Non-ME + NBI/BLI) 90% (Non-ME + WLI) 98% (ME)	63% (Non-ME + NBI/BLI) 76% (Non-ME + WLI) 56% (ME)	77% (Non-ME + NBI/BLI) 81% (Non-ME + WLI) 77% (ME)	NA	NA	Equivalent	Ohmori et al[23], 2020
Diagnosis	Retrospective	Still image	ME-NBI	FCN-CAD	1383 lesions	3-fold-CV	NA	87% (lesion level)	84.1 (lesion level)	89.2 (lesion level) 93.0 (pixel level)	NA	NA	Equivalent	Zhao et al[24], 2019
Diagnosis	Retrospective	Still image	ME-NBI	CNN	7046 images	5-fold-CV	NA	89.3%	98%	93.7%	NA	NA	NA	Everson et al[25], 2019
Diagnosis	Retrospective	Still image	ECS	CNN-GoogLeNet	4715 images	Caffe DL framework/1520 images	0.85	92.6%	89.3%	90.9%	NA	NA	NA	Kumagai et al[27], 2019
Invasion depth measurement	Retrospective	Still image	WLI/NBI/BLI	CNN-SSD	14338 images	Caffe DL framework/914 images	NA	90.1%	95.8%	91.0%	99.2%	63.9%	Equivalent	Nakagawa et al[28], 2019
Invasion depth measurement	Retrospective	Still image	WLI/NBI	CNN-SDD-GoogLeNet	1751 images	Caffe DL framework/291 images	NA	84.1%	73.3%	80.9%	NA	NA	Superior	Tokai et al[29], 2020
Diagnosis	Retrospective	Still image/Real-time video	NBI	CAD-SegNet	6473 images	6671 images/80 videos	0.989	98.04% (per-image) 91.5%(per-frame)	95.03% (per-image) 99.9%(per-frame)	NA	NA	NA	NA	Guo et al[30], 2020
Diagnosis	Retrospective/ Prospective	Still image/Real-time image	WLI	GRAIDS/DeepLab V.3+	4091 images	3323 images	NA	NA	NA	NA	NA	NA	Equivalent	Luo et al[31], 2020
Detection/Invasion depth measurement	Retrospective	Still image/Real-time video	NBI/BLI	CNN-SSD	17274 images	5277 images/144 videos	NA	91.1%	51.5%	63.9%	46.1%	92.7%	Superior	Fukuda et al[32], 2020
Invasion depth measurement	Retrospective	Still image/video images	WLI/NBI/BLI	CNN-SSD	23977 images	PyTorch DL framewor/102 video images	NA	50% (non-ME) 70.8%(ME)	98.7% (non-ME) 94.9%(ME)	87.3% (non-ME) 89.2%(ME)	92.3% (non-ME) 81.0%(ME)	86.5% (non-ME) 91.4%(ME)	Superior	Shimamoto et al[33], 2020

Although no detailed information about the classification of esophageal cancer (EC) was found in the research of Liu et al[20] and Luo et al[31], I prefer to summarize the data in this table because most EC patients in China suffer from esophageal squamous cell carcinoma. AI: Artificial intelligence; AUC: Area under the curve; SEN: Sensitivity; SPE: Specificity; ACC: Accuracy; PPV: Positive predictive value; NPV: Negative predictive value; HRM: High-resolution microendoscopy; 2-class LDA: Two-class linear discriminant analysis; NA: Not available; WCE: Wireless capsule endoscopy; JDPCA: Joint diagonalization principal component analysis; CCV: Color coherence vector; n-fold-CV: n-fold cross-validation; WLI: White light imaging; NBI: Narrow-band imaging; CNN: Convolutional neural network; SSD: Single-Shot Multibox Detector; DNN: Deep neural network; DL: Deep Learning; BLI: Blue Laser Imaging; ME: Magnifying endoscopy; FCN-CAD: Full convolutional network computer-aided detection; ECS: Endocytoscopic system; GRAIDS: Gastrointestinal Artificial Intelligence Diagnostic System.

ESCC diagnosis

In 2015, Shin et al[18] developed a class 2 Linear classification algorithm to identify squamous high-grade dysplasia (HGD) or ESCC by using nuclear-related features of high-resolution microendoscope (HRM) images. The area under the curve (AUC), SEN, and SPE of the test and validation data sets were 0.95, 87%, and 97% and 0.93, 84%, and 95%, respectively. In order to reduce equipment costs, Quang et al[19] developed a small tablet-interfaced HRM with real-time algorithms. The algorithm can automatically identify ESCC, and its AUC, SEN, and SPE were 0.937, 95%, and 91%, respectively. It was believed to be especially beneficial in lower-resource settings for operators with less experience interpreting HRM images.

In 2016, Liu et al[20] designed an algorithm called Joint Diagonal Principal Component Analysis (JDPCA) for detection of EC. In their research, a novel image feature extraction method was established through combining the algorithm of ML based on JDPCA and conventional feature extraction algorithm without learning. Then a computer-aided method was proposed to identify the endoscopic images obtained from conventional gastroscopy and wireless capsule endoscopy containing lesions. The algorithm can correctly detect 90.75% EC, and the AUC reached 0.9471 as a result. Although ESCC is the main pathological type in China, the specific classification of EC was not mentioned in the study.

Horie et al[21] first tried to use DL to diagnose ESCC through a large number of endoscopic images in 2019. The CNN took 27 s to analyze 1118 test images captured using WLI and NBI and correctly detected EC cases with a SEN of 98%. It could distinguish superficial EC from advanced cancer with ACC of 98%. Subsequently, Cai et al[22] proposed a novel computer-aided diagnosis (CAD) system using a deep neural network (DNN). Two thousand four hundred twenty-eight (1332 abnormal, 1096 normal) esophagoscopic images from 746 patients were collected to set up a novel DNN-CAD system in two centers, and a validation dataset containing 187 images from 52 patients was prepared. Only standard WLIs were used to train the model. The DNN-CAD model can detect 91.4% of early ESCC, which is higher than that of advanced endoscopists.

Ohmori et al[23] evaluated ME and non-ME images [including WLI and NBI/Blue Laser Imaging (BLI)] using Single-Shot Multibox Detector (SSD) based CNN to identify ESCC. The ACC of ME, non-ME+WLI, and non-ME+NBI/BLI was high SEN and moderate SPE, respectively, 77%, 81%, and 77%. It was proved in the study that no significant difference existed in diagnostic performance between AI and experienced endoscopists.

Endocytoscopic diagnosis

Zhao et al[24] conducted another CAD model to evaluate the feasibility of automated classification of IPCLs to improve the detection of ESCC. ME-NBI images were collected, and a double-labeled full convolutional network was developed for image segmentation. In the research, it was shown that the diagnostic ACC of senior observers was much higher than that of mid-level and junior observers. When this model distinguished the lesions A, B1, and B2 IPCL, the diagnostic ACC rates of the lesion and pixel level reached 89.2% and 93%, respectively, similar to the seniors. Specifically, the model was more sensitive to type A IPCL than clinicians (71.5% vs 28.2%-64.9%), which can avoid unnecessary radical treatment. A total of 7046 sequential HD ME-NBI images from 17 patients were used to train a CNN in another study conducted by Everson et al[25]. IPCL patterns were also classified according to the JES classification. This CNN differentiated abnormal from normal IPCL patterns with 93.7% ACC.

The endocytoscopic system (ECS) provides magnified endoscopy, which can be stained with methylene blue to observe surface epithelial cells[26,27]. The optical magnification of ECS is 500 ×, which can be increased to 900 × by using the digital magnification in the video processor[26]. Kumagai et al[27] used a CNN-based AI system based on GoogLeNet and trained using 4715 ECS images of the esophagus (1141 malignant and 3574 non-malignant images) to diagnose ESCC using ECS images. An independent set of 1520 images, collected from 55 consecutive patients (27 ESCCs and 28 benign esophageal lesions) were tested. The AI correctly diagnosed 25 of the 27 ESCC with an overall SEN of 92.6%, SPE of 89.3%, and an overall ACC of 90.9%.

Depth of invasion diagnosis of ESCC

The study conducted by Nakagawa et al[28] did not determine the IPCL pattern but aimed to predict the depth of cancer invasion by using a DL system based on the CNN-SSD architecture. For all, non-ME and ME images, the system's ability to distinguish correctly pathologic mucosal/submucosal microinvasive (EP/SM1) cancers from submucosal deep invasive (SM2/3) cancers was 91%, 92.9%, and 89.7%, respectively, with comparable performance to experienced endoscopists.

In Tokai et al[29] research, AI’s ability to measure the depth of ESCC invasion was demonstrated. They collected 1751 ESCC training images obtained by WLI and NBI from the Japan Cancer Institute Hospital and developed a CNN-SSD diagnosis system. The magnified images obtained via ME were excluded. Subsequently, 291 test images from 55 patients confirmed by pathological findings were used to compare the AI system with 13 board-certified endoscopists. The AI system detected 95.5% of the ESCC in the test images in 10 s and correctly estimated the ESCC infiltration depth in 6 s, with a SEN of 84.1% and an ACC of 80.9%. The ACC score of this system exceeded 12 of 13 endoscopists, and its AUC was greater than the AUC of all endoscopists. It was indicated that the AI system could potentially be used for ESCC diagnosis and measuring depth of invasion.

Real-time diagnosis of ESCC

Guo et al[30] developed a CAD system based on SegNet architecture that can perform real-time automatic diagnosis of precancerous lesions and early ESCC under non-ME and ME. Data from four medical centers in three countries including China, the United States, and India were collected and used for CAD model training. Four sets of data from early ESCC and noncancerous endoscopy including still images and real-time videos were used to verify their effects. The SEN and SPE of the system were 98.04% and 95.03%, respectively, for still images. In 27 non-magnifying videos, the per-frame and the per-lesion SEN were 60.8% and 100%, respectively. In 20 magnifying videos, the per-frame and the per-lesion SEN were 96.1% and 100%, respectively. Unaltered full-range normal esophagus videos included 33 videos (per-frame SPE 99.9%, per-case SPE 90.9%).

In another multi-center, case-control, diagnostic study, Luo et al[31] developed and validated the Gastrointestinal Artificial Intelligence Diagnostic System (GRAIDS) for the diagnosis of upper gastrointestinal cancers through analysis of imaging data from clinical endoscopies. Six hospitals of different levels with varying experience in the endoscopic diagnosis of upper gastrointestinal cancer in China participated in the study. The GRAIDS algorithm was based on the concept of DeepLab's V3+ [released by Google (Mountain View, CA, United States) in 2018]. In total, 1036496 endoscopic images from 84424 individuals were used to develop and test GRAIDS. GRAIDS achieved a diagnostic SEN similar to that of expert endoscopists {0.942 [95% confidence interval (CI): 0.924-0.957] vs 0.945 (0.927-0.959); P = 0.692}. The SEN was higher than that of qualified endoscopists [0.858 (0.832-0.880), P < 0.0001] and trainees [0.722 (0.691-0.752), P < 0.0001]. In this big data study, the number of EC in the training set and verification set reached about 27% of the total, but the proportion of early EC and the detection effect were not explained. More detailed data on early EC are to be expected.

The latest research results on real-time detection of early EC are from Fukuda et al[32]. They used 23746 images of 1544 cases of superficial ESCC confirmed by pathology and 4587 images of 458 cases of non-cancerous and normal tissues to construct a deep CNN-SSD model. NBI or BLI technology was used to collect 5-9 s video clips of 144 patients as a verification data set. The SEN, SPE, and ACC for the AI were 86%, 89%, and 88%, respectively, and 74%, 76%, and 75% for the experts, respectively. This is an important study that paves the way for the development of better models to detect EC lesions in a real-time clinical environment, but randomized prospective clinical trials are needed for verification.

Another research team from Japan developed an AI system to calculate ESCC invasion depth[33]. In total, 23977 images, including WLI and NBI/BLI images of pathologically proven ESCC from endoscopic videos and still images, were obtained as a learning dataset. An independent validation dataset of 102 video images of ESCC was taken in research. Two types of videos, non-ME with WLI and ME with NBI/BLI of 4-12s, were included. The CNN model was compared with 14 experts in endoscopic field. ACC, SEN, and SPE performance for AI were superior to those for experts in both types of videos. The AI model was concluded to be effective in real-time measurement of ESCC invasion depth.

AI IN ENDOSCOPIC DETECTION OF BE AND EAC

BE is the precursor of EAC. The prevalence of EAC is rising rapidly, and the prognosis of advanced cases is poor. However, early detection of cases can be promptly treated with endoscopic treatment, which has a chance of cure. Due to the difficulty in detecting early neoplasms under endoscopy, early lesions are often missed[34]. This is because early Barrett tumors are usually flat, with only subtle changes in the color and texture of the mucosa. These changes can be visible in the HD-WLI performed by experts. However, due to the low progression rate of tumor formation (< 1% per patient year), many general-level endoscopists rarely encounter the early formation of Barrett's tumor, so that they are not familiar with its endoscopic morphology and cannot recognize the lesions[35]. In order to improve EAC screening and increase the SEN and speed of the examination, AI-assisted endoscopy has important value. In addition, it can also provide convenience for endoscopists facing challenges and anxiety, so as not to miss early lesions (Table 2).

Table 2 Artificial intelligence application for esophageal adenocarcinoma.

AI Application	Study design	Data category	Type of Images	AI architecture	Training dataset	Validation Method or dataset	AUC	SEN	SPE	ACC	PPV	NPV	Compared with experts	Ref.
Detection	Retrospective	Still image	WLI	CAD-SVM	64 images	LOOCV	NA	95%	NA	75%	NA	NA	NA	van der Sommen et al[36], 2014
Detection	Retrospective	Still image	WLI	CAD-SVM	100 Images	LOOCV	NA	83% (per-image) 86% (per-patient)	83% (per-image) 87% (per-patient)	NA	NA	NA	Inferior	van der Sommen et al[37], 2016
Detection	Retrospective	Still image	VLE	CAD	60 images	LOOCV	0.95	90%	93%	NA	NA	NA	Superior	Swager et al[38], 2017
Detection	Retrospective	Still image	VLE	CAD	60 images	LOOCV	0.90-0.93	NA	NA	NA	NA	NA	Superior	van der Sommen et al[40], 2018
Diagnosis	Retrospective	Still image	WLI/NBI	CNN	8 patients	Caffe DL framework	NA	88% (WLI)/88% (NBI) (per-patient) 69% (WLI)/71% (NBI) (per-image)	NA	90%	NA	NA	NA	Horie et al[21], 2019
Detection	Retrospective	Still image	WLE	CNN-SSD	100 images/39patients	20% patients/5-fold-CV/LOOCV	NA	96%	92%	NA	NA	NA	NA	Ghatwary et al[41], 2019
Detection	Retrospective	Still image	WLI/NBI	CNN- Inception-ResNet-v2	1832 images	458 images	NA	96.4%	94.2%	95.4%	NA	NA	NA	Hashimoto et al[42], 2019
Detection	Retrospective	Still image	WLI	CAD	60 images	LOOCV	0.92	95%	85%	91.7%	NA	NA	NA	de Groof et al[43], 2019
Detection	Retrospective	Still image	WLI	CAD-ResNet-UNet	1544 images	4-fold-CV (internal validation)/160 images (external validation)	NA	87.6% (internal validation) 92.5% (external validation)	88.6% (internal validation) 82.5% (external validation)	88.2% (internal validation) 87.5% (external validation)	NA	NA	NA	de Groof et al[44], 2019
Diagnosis	Retrospective	Still image	WLI/NBI	CAD-ResNet	248 images	LOOCV	NA	97% (WLI)/94% (NBI) (Augsburg data)92% (MICCAI)	88% (WLI)/80% (NBI) (Augsburg data)100% (MICCAI)	NA	NA	NA	NA	Ebigbo et al[45], 2019
Diagnosis	Retrospective	Random images from real-time video	WLI	CAD-ResNet-/DeepLab V.3+	129 images	36 images (real time)	NA	83.7%	100%	89.9%	NA	NA	NA	Ebigbo et al[49], 2020
Surveillance	Prospective	Real-time image	WLI/NBI/VLE	IRIS	NA	Real-time image	NA	NA	NA	NA	NA	NA	NA	Trindade et al[50], 2019
Detection	Prospective	Live endoscopic procedure	Live endoscopic procedure	CAD-ResNet/U-Net	1544 images	48 levels/144 images/20 live endoscopic procedure	NA	90.9% (per level) 75.8% (per image) 90% (per patient)	89.2% (per level) 86.5% (per image) 90% (per patient)	89.6% (per level) 84.0% (per image) 90% (per patient)	NA	NA	NA	de Groof et al[51], 2020

AI: Artificial intelligence; AUC: Area under the curve; SEN: Sensitivity; SPE: Specificity; ACC: Accuracy; PPV: Positive predictive value; NPV: Negative predictive value; WLI: White light imaging; CAD: Computer-aided diagnosis; SVM: Support vector machines; LOOCV: Leave-one-out cross-validation; NA: Not available; VLE: Volume laser endoscopy; NBI: Narrow-band imaging; CNN: Convolutional neural network; DL: Deep learning; SSD: Single-Shot Multibox Detector; n-fold-CV: n-fold cross-validation; MICCAI: Medical Image Computing and Computer Assisted-Intervention; IRIS: Intelligent real-time image segmentation.

EAC diagnosis

In 2013, van der Sommen et al[36] first proposed a SVM algorithm for detecting early cancer in the EAC using HD endoscopic images, color histograms ,and simple statistical data of Gabor features, with a classification ACC of 95.9% and an AUC of 0.992. Later, they found that the system detected 36 from 38 lesions with a recall of 0.95 and a precision of 0.75 through clinical validation. In 2016, van der Sommen et al[37] tested a computer algorithm (automated image recognition system) that employed specific texture, color filters, and ML for detecting early neoplastic lesions on BE based on specific imaging details. All images were of high perceptual quality. The algorithm was developed and tested using 100 endoscopic images of 44 BE patients. Early tumor lesions were identified in the per-image analysis, with a SEN and SPE of 0.83. At the patient level, the SEN and SPE of the system were 0.86 and 0.87, respectively.

Swager et al[38] conducted the first study based on volume laser endoscopy (VLE) images with direct histological correlation in 2017, which developed a clinically inspiring computer algorithm for BE neoplasia[39]. VLE is an advanced imaging system that can scan the esophagus wall up to 3 mm deep with near-microscopic resolution. A total of 60 VLE images from a high-quality ex vivo VLE histological correlation database were used in the study, including 30 nondysplastic BE and 30 HGD/early EAC images. The algorithm of the feature "layering and signal decay statistics" showed good performance in detecting BE neoplasia (AUC = 0.95), and thus it might possible to assist endoscopists in detecting early neoplasia on VLE. van der Sommen et al[40] also compared the performance of the CAD methods with the classification ACC of two VLE experts, with a maximum AUC in the range of 0.90-0.93 for the ML method vs an AUC of 0.81 for the medical experts.

Horie et al[21] proved that the diagnosis ACC for EAC of the CNN was 90% (19/21). Four lesions of EAC were missed, which may be because educational images for EAC were not enough. Only eight were used in the study. In another study about CNN, four methods including Regional-based CNN (R-CNN), Fast R-CNN, Faster R-CNN, and SSD were adapted to detect abnormal regions in the esophagus HD-WLE images automatically[41]. A total of 100 images from 39 patients that have been manually annotated as ground truth by five experienced clinicians were tested. The SSD outperformed other methods achieving SEN of 0.96, SPE of 0.92, and F-measure of 0.94. Hashimoto et al[42] conducted a pilot study on the endoscopic detection of early esophageal neoplasia on BE using a CNN system. They pre-trained the system on ImageNet, then received training on 1853 images, and finally tested it with 458 images. The size of the tumor lesions in the image data they selected was between 3 mm and 20 mm, and most of the lesions were not significant, so that endoscopists with less training were likely to miss the diagnosis. The CNN accurately detected 95.4% of early lesions, including 96.4% of SEN and 94.2% of SPE, and the running speed of the algorithm was allowed for real-time implementation.

Progress on CAD development was made by de Groof et al[43] for BE detection. All WLIs recorded in 40 cases of neoplastic Barrett lesions and 20 cases of non-dysplastic BE were collected through WLE in full HD format (1280 × 1024 pixels). Experts delineated the images of neoplasm. The CAD system had received training in color and texture features. The ACC, SEN, and SPE for detection were 92%, 95%, and 85%, respectively, which were evaluated by leave-one-out cross-validation. In the randomized controlled trial (No: NTR7072.) reported in 2019, de Groof et al[44] developed a hybrid ResNet-UNet model CAD system using five independent endoscopy data sets. Compared with general endoscopists, the ACC of the CAD system was 88% vs 73%, the SEN was 93% vs 72%, and the SPE was 83% vs 74%. This report details the first externally validated DL-CAD system for early BE neoplasia.

In the study of Ebigbo et al[45], two databases [Augsburg data and the ‘Medical Image Computing and Computer Assisted-Intervention’ (MICCAI) data] were used to train and test a CAD system on the basis of a CNN with a residual net (ResNet) architecture. Based on still images, the diagnosis of EAC by CAD-DL reached SEN/SPE of 97%/88% (Augsburg data) and 92%/100% (MICCAI data) for WLIs and 94%/80% for NBIs (Augsburg data), respectively.

Depth of invasion diagnosis of EAC

Several studies questioned whether surgery should always be used to treat EAC with deep submucosal invasion. However, it is accepted that the depth of invasion is at least one of the biggest risk factors for metastasis[46]. In the searching item related to the word “depth of invasion/ infiltration of EAC”, no article published was found. Fortunately, research on AI's infiltration depth measurement of gastric cancer and colon cancer has been published, which may be worth learning for research of invasion depth of EAC. In the study of Zhu et al[47], a total of 790 WLIs served as a development dataset and another 203 images as a test dataset, and a CNN-CAD system, developed from ResNet50, was used to determine the invasion depth of gastric cancer. At a threshold value of 0.5, the SEN was 76.47%, SPE was 95.56%, and the overall ACC was 89.16%. The AI system achieved significantly higher ACC than human endoscopists. Scholars from Japan analyzed endocytoscopic images by using CAD-SVM, and the algorithm obtained after extracting 5543 images from 238 lesions can diagnose invasive colorectal cancer in 200 test images with an ACC of 94.1%[48].

Real time diagnosis of EAC

Ebigbo et al[49] developed a real-time DL system based on deep CNN and ResNet architecture with the latest encoder/decoder network DeepLab V.3+. The endoscopy was performed by an expert endoscopist. The system captured random images from real-time camera livestream and provided global prediction (classification) and dense prediction (segmentation), which can accurately distinguish between normal BE and early EAC. The ACC of the AI system for 14 cases of early EAC was 89.9%. The AI-based CAD was indicated to improve the quality of BE assessment, especially for non-expert endoscopists.

In 2019, Trindade et al[50] reported an AI software termed intelligent real-time image segmentation for endoscopic surveillance of BE. The intelligent real-time image segmentation identified three established VLE features previously associated with histologic dysplasia and displayed them using different color schemes superimposed over the VLE image. A multi-center randomized controlled trial (NCT03814824) is underway to validate further the AI system. In another study, de Groof et al[51] evaluated the preliminary diagnostic ACC of a CAD system in detecting Barrett's neoplasm during live endoscopic procedures. The test dataset composed of the live endoscopy procedure of 10 patients with nondysplastic Barrett's esophagus and 10 patients with confirmed Barrett's neoplasm. WLIs at every 2 cm level of the Barrett’s segment were obtained and analyzed. The ACC, SEN, and SPE of the CAD system were 90%, 91%, and 89%, respectively. This result revealed that the CAD system was ready to be tested in larger multi-center trials.

LIMITATION OF AI APPLICATION IN EARLY EC DETECTION

AI is becoming rapidly integrated into modern gastroenterology with the expansion of diagnostic images and treatments. Among the three common tumor diseases of the digestive tract, esophageal cancer, gastric cancer, and colon cancer, the value of AI has been fully demonstrated. Multiple AI algorithms have been developed to run real-time during gastroscopy and colonoscopy for cancerous detection, diagnosis, or invasion depth measurement[48,52,53]. The application of AI in the field of gastric cancer and colon cancer seems to be earlier and more comprehensive than that in the field of esophageal cancer.

It can be seen from the above review that the clinical application of AI in early EC is still in its infancy. The vast majority of studies are retrospective studies, and only two clinical trials have been reported, one of which is still in research progress. The above-mentioned research mainly focused on AI algorithms, training and verification and compared with the detection ACC, SEN, and SPE of different levels of endoscopists. There has not been a large-scale clinical application of AI, whether for surveillance or detection. It is foreseeable that the difficulties in applying AI to clinical practice are manifold, and there are still many obstacles before large-scale implementation.

First of all, the current AI applications mainly focus on files such as images, pictures, and videos. The establishment of an AI model requires a sufficiently large training set labeled by experts for AI learning. This training set requires the participation of a number of expert-level staff, and experts are obviously scarce resources, which makes it difficult for the quantity and quality of the training set to meet sufficient demand. As natural persons, experts are also likely to make mistakes. This may lead to incorrect diagnostic information in the training set. Secondly, a large amount of medical imaging data, especially endoscopy data related to EC, need to be obtained during manual examination. The quality of the data depends not only on the level of the endoscopist but also on the function of the endoscopic instrument. Not all endoscopists can complete a qualified endoscopy, and not all endoscopy equipment can provide the best images. Most medical units can only use their own data, which can be considered a selection bias. AI must be applied to external multi-centers for validation to explore its ACC. At present, only a multi-center study from China had conducted external verification[31]. Third, the subject of AI use in early EC detection is currently mainly focused on retrospective research. It is difficult to use AI technology for prospective clinical research, especially blinded randomized controlled trials, although this type of research is the most reliable "gold standard" in medical science. To conduct such a trial, it requires medical ethics approval. Fourth, the current generation of AI is numerous but proprietary. Costs including equipment and person training may become a limiting factor for widespread implementation. Fifth, the acceptance and approval of AI by governmental and professional organizations are needed. For each country, any legal issues related to the use of AI in endoscopes must be resolved before widespread implementation.

FUTURE OF AI APPLICATION IN EARLY EC DETECTION

Papers on AI in the diagnosis of early EC are gradually increasing, but except for a few, they basically discuss AI algorithms and clinical validation. The training data and verification data used are also still images. In the study on EAC , the authors also used still images for real-time analysis during endoscopy[49], which may be useful for evaluating EAC and other situations because the area of BE is limited. It is more useful to use real-time video for detection, because the detection can be performed immediately without the need to capture pictures frequently. In order to investigate whether AI is useful for preventing missed cancerous diagnosis, validation videos should not be limited to slow observation videos but include fast videos. However, this high accuracy comes from high-resolution videos, and it is not easy to provide accurate and high-quality images in videos containing low-resolution. As a guarantee of computing speed, the improvement of graphics processing unit and other hardware needs to be synchronized.

Existing papers discuss the still images of the esophageal surface and the depth of invasion, indicating that endoscopic equipment can not only improve the image quality but also become three-dimensional. When combined with other sensing technologies, it is eager to perform three-dimensional mapping of esophageal lesions through an endoscope. If AI technology is combined with three-dimensional endoscopy, what will happen in the future? There is no ending.

The AI system is expected to become a qualified assistant, but not a substitute, for inexperienced endoscopists to get similar judgment capabilities as experts. In order to establish an AI training database applicable to the whole world, it is necessary to establish a gold standard for AI diagnosis. This requires endoscopy experts from different countries and regions to label accurately and supplement the database.

Even though AI is helpful for diagnosis during endoscopy, it is temporarily difficult for AI to perform endoscopy operations on behalf of humans because the shape of the digestive tract varies from person to person. Therefore, for endoscopists, efforts to improve the technology used to operate endoscopes will continue to be indispensable.

CONCLUSION

This review summarizes the current status of research and development of AI in endoscopes related to early EC detection and the application prospects of AI. The use of AI-based endoscopes will enable detection accuracy of early EC and improve prognosis. As the diagnosis ability and accuracy of early EC vary greatly among endoscopists, AI may have better diagnostic capabilities and will help endoscopists to reduce missed diagnosis of early EC in the near future. Maybe only a step is needed to move for AI application in such a clinical field, but the step needs efforts from multiple fields including medicine, AI, optics, legislation, ethics, etc. We might as well look forward to this step as soon as possible.