Artificial intelligence in gastrointestinal radiology: A review with special focus on recent development of magnetic resonance and computed tomography

Abstract

Artificial intelligence (AI), particularly the deep learning technology, have been proven influential to radiology in the recent decade. Its ability in image classification, segmentation, detection and reconstruction tasks have substantially assisted diagnostic radiology, and has even been viewed as having the potential to perform better than radiologists in some tasks. Gastrointestinal radiology, an important subspecialty dealing with complex anatomy and various modalities including endoscopy, have especially attracted the attention of AI researchers and engineers worldwide. Consequently, recently many tools have been developed for lesion detection and image construction in gastrointestinal radiology, particularly in the fields for which public databases are available, such as diagnostic abdominal magnetic resonance imaging (MRI) and computed tomography (CT). This review will provide a framework for understanding recent advancements of AI in gastrointestinal radiology, with a special focus on hepatic and pancreatobiliary diagnostic radiology with MRI and CT. For fields where AI is less developed, this review will also explain the difficulty in AI model training and possible strategies to overcome the technical issues. The authors’ insights of possible future development will be addressed in the last section.

Key Words: Artificial intelligence; Deep learning; Image diagnosis; Radiology; Magnetic resonance imaging; Computed tomography

Core Tip: Gastrointestinal radiology is a subspecialty that is important and complex, and is thus a popular subject in artificial intelligence (AI). Recently many deep-learning based diagnosis assistance tool have been developed in gastrointestinal radiology, particularly in diagnostic abdominal magnetic resonance imaging (MRI) and computed tomography (CT). Herein we will review recent advance of AI in gastrointestinal radiology, with a special focus on abdominal MRI and CT. Current difficulty in less-developed fields will be explained as well.

INTRODUCTION

The field of gastrointestinal radiology includes diagnostic radiology and interventional radiology. In the practice of diagnostic gastrointestinal radiology, various imaging tools are applied for the diagnosis of lesions in the abdominal cavity. These tools include X-ray used in abdominal plain film[1], angiography and abdominal computed tomography (CT)[2], magnetic resonance used in abdominal magnetic resonance imaging (MRI)[3,4], and ultrasound used in abdominal sonography[5]. For some diagnostic tasks, intravenous contrasts are used to enhance lesions for study. Contrast-enhanced, three-phase CT is the standard for examination of liver tumors and many other lesion types[6]. Contrast-enhanced ultrasound and MRI, though less frequently used, have some clinical use in examination of pancreatic lesions and inflammatory bowel disease[7-9]. Please refer to Ripollés et al[7] for example of contrast-enhanced ultrasound for diagnosis for Crohn’s disease.

Artificial intelligence (AI) have been influential in radiology recently, because it has potential to reduce workloads of radiologists, and diagnostic radiology tools stated above have provided feasible ground for machine learning model development. Potential of machine learning models to reduce radiologist workload come from its better stability, higher work efficiency, and better accuracy in some selected tasks[10] than human workers. Deep learning has proven its suitability for different imaging methods, and radiology and has been widely used in image classification, segmentation, detection, and reconstruction tasks[11]. There are some optimistic radiologists who are willing to let AI assist them in their work so that they can enhance their role in other places[12,13]. Of course, there are also pessimistic radiologists who worry that the development of AI systems will replace radiologists[14].

The most significant shortcoming of machine learning algorithms require a lot of data[15]. At the same time, the lack of unified standard training data will lead to a decrease in the efficiency of AI learning, but it is difficult for doctors to label a large amount of accurate data in complex diseases. In addition, the algorithm may learn false correlations, which may also lead to overfitting. At the same time, it is difficult for AI to explain the causality in the observation dataset. Semi-supervised learning is between supervised learning and unsupervised learning. In the training process, a small amount of labeled data and a large amount of unlabeled data are used at the same time. The development of semi-supervised learning algorithms is mainly because data labeling is very expensive or impossible in some fields[16-18]. The development of semi-supervised learning can also simultaneously solve the problems of a large number of labeling and overfitting.

INTERVENTIONAL RADIOLOGY

Interventional radiology uses imaging techniques in diagnostic radiology to treat diseases or take specimens. The practice of interventional procedures in gastrointestinal radiology can be best exemplified by the treatment solid organ tumors. Among the most-frequently used non-surgical treatment procedures of hepatocellular carcinoma (HCC) are transcatheter arterial chemoembolization (TACE) and radiofrequency ablation (RFA). In TACE[19], liver tumors are first highlighted by angiography, and then embolized by particles coated with chemotherapeutic drugs. In RFA[20], the lesion is located by ultrasound rather than angiography and ablated by radiofrequency heating. In addition to liver cancer, any solid organ tumors with rich vasculature can be treated with this procedure. For example, pancreatic neuroendocrine tumors are frequently hypervascular, therefore are sometimes treated by embolization since last century[21,22], especially in patients with multiple endocrine neoplasia type 1 syndrome, where multiple tumors may make resection unfeasible[23]. are also widely applied in some of pancreatic tumors, such as neuroendocrine tumors. Application of RFA, which does not require rich vasculature, is even more versatile than TACE. There are reports of successful radiofrequency ablation on unresectable pancreatic cancer[24,25], and even intra-abdominal sarcomas such as gastrointestinal stromal tumor[26].

Interventional radiology also has broad application on non-tumor diseases, especially in vascular diseases. The best well-known example is emergent management of gastrointestinal bleeding, where the bleeding artery can be visualized by angiography, and embolized[27,28]. A similar approach can be also applied to thrombotic diseases such as Budd-Chiari syndrome or celiac artery occlusion[29,30]. In management of these disorders, the vessels are visualized and dilated with stents or dissolved with thrombolytic agents. Applications of interventional radiology are numerous and still developing, so a thorough review is out of scope of this article.

Both diagnostic and interventional gastrointestinal radiology can be done endoscopically. For example, in endoscopic ultrasound (EUS), the ultrasound probe is inserted through an endoscope to visualize lesions that are not easily accessible by abdominal sonography[31,32]. Biopsy and other interventional procedures can then be done to the visualized lesion via the endoscope, as exemplified in publications by Williams et al[33]. and Kahaleh et al[34]. Endoscopic radiological images are more difficult to be collected in large amount, because like in EUS, most image from endoscopic procedures are manually captured with custom angle of the endoscopist, rather than in an automatic and standard manner. Therefore, unlike in development of AI in regular diagnostic radiology, in which large scale public dataset, such as pancreas CT dataset from The Cancer Imaging Archive[35-37], and Beyond the Cranial Vault Abdomen data set[38,39], are readily available, most AI studies in endoscopic radiology still requires collection and processing of multihospital data. Moreover, lack of standardization and technical difficulty can make researchers reluctant or afraid to make image public. For example, in the study of computer-aided diagnosis of gastrointestinal stromal tumors by Li et al[40], the authors made the research possible only after collecting data from 19 hospitals, and did not publish the dataset. To our knowledge, there is only one well-known, public database of endoscopic ultrasound, published in 2020[41], and we hope that more database will be available in the following decade. In the present situation, due to less available resource, endoscopic radiology is less developed, so in this review article, we will focus on non-endoscopic radiological examination, particularly on CT and MRI.

DEEP LEARNING IN RADIOLOGY: ACHIEVING STATE OF THE ART IN LESION DETECTION

In the last five years, there have been marked progress in deep learning-assisted lesion detection for radiology, particularly in computed tomography. The progress can be exemplified by the DeepLesion tool developed by National Institutes of Health[42], which claims to detect all types of lesion in computed tomography regardless of the organ, with a sensitivity of 81.1% and five false-positives per case. DeepLesion was published along with an immense dataset with 32120 CT slices. With this annotated database in hand as a powerful tool, researchers refined lesion detection algorithm at an accelerated pace. For example, with the DeepLesion dataset, researchers from Chinese Academy of Sciences were able to develop the MVP-Net tool[43] by feature pyramid network, which claims to be 5.65% more sensitive than DeepLesion. With more developed advancements in deep learning algorithms and more databases available, we can expect that universal lesion detection in computed tomography will reach clinical use in reasonable time. An example of lesion detection in DeepLesion can be found in Yan et al[42].

For MRI, recent advancements are much less pronounced. Due to complex and variable sequencing techniques used in MRI, such as perfusion weighted imaging and T2* used in in stroke protocol[44] and diffusion weighted imaging[45] used in various organs, development of an universal, organ-neutral lesion detection algorithm is very difficult, if not impossible. Nonetheless, for individual organs, there is still marked progression. For example, using a deep learning algorithm, Amit et al[46] developed a tool for lesion detection in breast MRI. Later, in 2019, with the application of deep learning on T1-weighted, fat-suppressed MR images, Kijowski et al[47] further extended the technology to predict breast lesion type. Though not as effective as in breast lesion detection, the application of deep learning on musculoskeletal system MRI has achieved marked success for the detection of variable lesions, such as fracture, deformity, and metastatic disease. There are numerous studies about lesion detection on MRI in other organs, but it is beyond the scope of this review article.

Given the fact that there are on an average five false-positive lesions detected by DeepLesion, deep learning algorithms trained by radiographs are prone to over-detecting lesions. Researchers are aware of this problem and have tried to overcome it by various technologies. The most-used and earliest method applied is multi-view convolutional networks (CNN), wherein native 3D shapes are recognized from their rendered 2D views[48]. By using multi-view CNN, Setio et al[49], Kang et al[50] and El-Regaily et al[51] reported significant reduction of false-positive lesions in the lung with computed tomography, thus making this algorithm the most effective detection training tool for lung image. Recent results of the use of multi-view CNN in lung lesion detection are shown in Table 1.

Table 1 Recent results in usage of multi-view convolutional networks in lung lesion detection.

Dataset	Toolset	AUC	Ref.
LIDC	ConvNets (2D)	0.996	Setio et al[49], 2016
LIDC	Inception-Resnet (3D)	0.99	Kang et al[50], 2017
LIDC	MatConvNet (2D)	0.94	El-Regaily et al[51], 2020

LIDC: Lexington Infectious Disease Consultants; AUC: Area under the curve.

In addition to lung computer tomography, multi-view CNN has been used with other imaging subjects as well. It is also used to increase specificity in mammographic image classification[52] and longitudinal multiple sclerosis lesion segmentation[53]. Besides multi-view CNN, masking techniques during neural network training are also used to reduce false positive lesions. For example, Zlocha et al[54] used dense masks to improve the performance of RetinaNet[55], and the researchers developing ULDor tool[56] used pseudo mask to reduce false positivity in universal lesion detector.

Taken together, in recent years, deep learning for lesion detection in technology has shown great progress. In the next section, we will focus on how these technical advancements have benefited the diagnosis of gastrointestinal lesions.

DISEASE DIAGNOSIS AND PREDICTION IN GASTROENTEROLOGY

Cholangiographic diagnosis

One of the most advanced achievement in gastrointestinal radiology is the non-invasive evaluation of for the bile ducts. Before the era of image reconstruction and advanced endoscopy, visualization and diagnosis of lesions causing biliary disease usually required quite invasive procedures such as transhepatic cholangiography[57]. In late 20^th century, with the advancements in endoscopy, it was replaced by endoscopic methods like retrograde cholangiopancreatography (ERCP)[58] and EUS cholangiography[59,60]. For achieving both treatment and diagnosis, endoscopic procedure maybe necessary and appropriate, but for the sole purpose of diagnosis, such as visualization of lesions in primary sclerosing cholangitis (PSC)[61] and choledochal cyst[62], endoscopic procedure maybe too invasive and inconvenient for patients.

Therefore, in the last three decades, with the increasing demand of non-invasive procedures and the progress of digital image reconstruction technologies, some radiology visualization tools, such as magnetic resonance cholangiopancreatography (MRCP)[63] and CT cholangiography[64], have been developed and achieved clinical importance. For diagnostic problems, the precision of non-invasive examination has become comparable to that of endoscopic procedure. MRCP achieved diagnostic accuracy of up to 97% in the diagnosis of choledocholithiasis as early as 2000[65]. In 2011, MRCP even rivaled the performance of pathologic examination, with an accuracy of 82.9% in predicting carcinomatous biliary obstruction[66]. In the meantime, CT cholangiography also reached the status of standard care in some situations, such as preoperative biliary anatomy assessment when MRCP is inconclusive[67].

These noninvasive diagnostic examinations are, of course, far from perfect. Despite early success, in some studies between 2010 and 2020, the sensitivity of MRCP for choledocholithiasis was reportedly inferior to that of EUS[68]. This outcome may be attributed to subjectivity and inter-observer variability of interpretation, because, even though it is less demanding than ERCP, the radiological assessment of the bile duct and pancreas still requires high level of expertise to interpret[69]. For more demanding tasks, such as detection and classification of pancreatic lesions[70,71], the performance of noninvasive tests can be even more disappointing.

To cope with the problem of interpretation difficulty in noninvasive cholangiopancreatography, researchers began to use variable deep learning methods in an attempt to achieve more subjective and sensitive lesion detection in the bile ducts and pancreas. For example, Ringe et al[72] developed a transfer learning-based system for automated detection of PSC, achieving a sensitivity of 95%. If this system is used clinically, radiologists can avoid all-manual interpretation for difficult PSC detection, thus reducing possible the inter-observer disagreement. Some of researchers also used deep learning to improve image reconstruction and segmentation in the pancreatobiliary region, to reduce pitfall in traditional MRCP and CT cholangiography. For example, Tang et al[73] used deep learning to improve highlighting of periampullary regions in MRI, which can be difficult with traditional MRCP method. Al-Oudat et al[74] used Denoising Convolutional Neural Networks for better construction of intrahepatic biliary segmentation in MRI image.

Besides its utility in noninvasive examination, deep learning can also benefit imaging difficulty in endoscopic procedure. By a segmentation algorithm trained by D-LinkNet34 and U-Net, Huang et al[75] developed a system to evaluate stone removal difficulty of ERCP. By training on a deep learning model using ultrasound images and videos, Zhang et al[76] developed a system to recognize pancreas segments and stations in EUS. With globally increasing computing power and maturing deep learning technology, we can expect radiological pancreaticobiliary system assessment to continuously improve in the future.

Detection and classification of solid organ tumor

Imaging studies, such as abdominal contrasted CT scan and contrast enhanced ultrasound, are crucial for the evaluation of solid organ tumor diagnosis, such as liver cancer, pancreatic cancer, and other solid organ tumors. The best example is screening for HCC in patients with cirrhosis[77]. Image diagnosis of liver tumor is crucial and effective to the point that HCC can be diagnosed by three-phase contrasted CT[78] alone, without the need of a biopsy[79]. Despite being less accurate, image diagnosis is helpful in more difficult-to-diagnose tumor types, such as focal nodular hyperplasia and hepatocellular adenoma[80-82]. CT diagnosis is also crucial and sensitive for pancreas cancer diagnosis[83] and prediction of malignant change in cystic lesion[84].

The first problem in image diagnosis is that, even with state-of-the-art, highly sensitive technique, it can have less than ideal specificity. For example, image appearance of intrahepatic cholangiocarcinoma (ICC) can mimic HCC both in contrast-enhanced CT[85] and contrast-enhanced ultrasound[86]. Since the long-term outcome and treatment strategy are significantly different between HCC and ICC[87,88], this can be a severe misdiagnosis that impacts prognosis. Some vascular tumors like epithelioid hemangioendothelioma[89,90] and sclerosed hemangioma[91] may also mimic epithelial malignancy, making the image diagnosis even less specific. Moreover, because of a large volume of abdominal CT and MRI done for liver cancer screening, the workload is quite a lot for radiologists[92,93]. Pancreatic cancer is more problematic, since inflammatory process such as autoimmune pancreatitis can mimic adenocarcinoma, causing diagnostic difficulty in CT and MRI[94,95]. Less prevalent tumor types, such as acinar cell carcinoma of pancreas, can be even more challenging[96]. Therefore, there is strong demand for automatic tumor classification algorithm for abdominal imaging, to improve the accuracy of tumor classification and reduce radiologists’ workload.

Of the two purposes stated above, the most recent development was on assisted lesion detection to relieve radiologists’ workload. Using watershed transform and Gaussian mixture, Das et al[97] developed a tool that they claimed can detect hemangioma, HCC and metastatic carcinoma with a classification accuracy of 99.38%; however, they did not consider ICC in their differential diagnosis, therefore, this tool can be used only for screening, and not for final tumor diagnosis. Vorontsov et al[98] used fully convolutional network for the detection of liver metastatic colorectal cancer, with a sensitivity of up to 85%. There are several other developed for liver tumor detection and segmentation with variable success[99,100]. For automatic pancreatic cancer detection, there are also variable success. Li et al[101] developed a computer aided diagnosis model by Dual threshold principal component analysis for pancreas cancer on PET/CT image, with an accuracy of up to 87.72%. By using faster region-based CNN on CT image, Liu et al[102] built a diagnosis system which detected pancreatic cancer with an area under the curve (AUC) of 0.9632. These studies are only some examples of AI detection of digestive system cancer in medical images. For a more detailed discussion, readers can refer to the other review article focused on this subject[103].

Few researchers have published results about detailed tumor classification based on abdominal imaging. By training convolution CNN with both MRI image and clinical data, Zhen et al[104]’s model achieved AUC of up to 0.985 in the classification of malignant tumors as hepatocellular carcinoma, metastatic carcinoma or other primary malignancies. Yasaka et al[105] attempted automatic classification of liver tumor into five classes (HCC, other malignancy, indeterminate masses, and two classes of benign lesions) using CNN, and achieved an accuracy of 0.84. Scope of these classification tools are summarized in Table 2. Due to limited literature available, it is too early to predict whether automatic radiological tumor classification will be comparable to pathologic diagnosis, but the recent results seem promising, and would be a good subject for further research.

Table 2 Classification scope for recent deep learning-based tumor classification tools.

Ref.	HCC	ICC	Metastatic carcinoma	Other malignancy	Benign tumors
Das et al[97], 2019	O	X	O	X	O
Vorontsov et al[98], 2019	X	X	O	X	X
Zhen et al[104], 2020	O	O	O	O	X
Yasaka et al[105], 2018	O	O	O	O	O

HCC: Hepatocellular carcinoma; ICC: Intrahepatic cholangiocarcinoma. X: Yes; O: No.

Intelligent assistance on endoscopic radiology

Endoscopic radiological procedures, such as EUS and ERCP, can be very difficult to perform and interpret, and require a lot of training to achieve competence[106], particularly if combined with interventional procedures like ampullectomy or biopsy[107,108]. Artificial intelligence assistance to reduce difficulty and allow for a reasonable learning curve is therefore desired for these procedures.

Due to the limited availability of public image database of EUS and ERCP, the development of AI models for these modalities is, as stated in a previous review article, still in its infancy[109]. There are, however, already some promising results in assistance of endoscopic radiological procedure. The most pronounced progress is with depth assessment in EUS. EUS imaging for evaluation of tumor depth is crucial in predicting the safety of endoscopic submucosal dissection[110]; however, the image diagnosis can be subjective, and requires much expertise. Cho et al[111] developed a tool using deep learning that predicts tumor depth in EUS with a claimed AOC of 0.887. For less sophisticated tasks such as detection of pancreatic cancer in EUS, the result is even better, with a claimed AOC of 0.940[112]. Therefore, it is evident that deep learning-assisted diagnosis can be a reliable tool.

In summary, AI has proven helpful in radiological diagnosis. Although few of the tools described above have reached clinical use, with current development, we can expect AI-assisted diagnosis to advance further in few years, and it may eventually become relevant to everyday clinical practice.

MAIN CHALLENGES AND PITFALLS OF THE APPLICATION OF AI IN RADIOLOGY

Although AI has made a lot of contributions in radiology, there are still some challenges and pitfalls, and AI experts should be cautious when working with radiologists. One of the biggest challenges is the availability of data. Ordinary deep learning algorithms will be learned through millions of training datasets, but it is difficult for the medical field to have such a large amount of data, and even if there are a large number of training datasets, there is currently no unified classification standard[113,114]. If the training dataset is too small, multiple neuron training through deep learning will easily lead to overfitting[115,116] and will show poor accuracy in independent tests. How to choose the right amount of model depth to adapt to a smaller training dataset will be the biggest challenge for AI engineers. In addition, generative adversarial networks[117] is also very suitable for small training datasets. At the same time, the establishment of a large number of training databases can also effectively help improve the efficiency of AI. Physicians and engineers work together to establish an open database and set uniform standards, which can also enhance AI applicability in radiology and pathology.

In addition, some diseases (usually rare diseases) have a problem of extreme disparity in the classification ratio, which is called imbalanced data. Imbalanced data training is more difficult, which usually leads to high accuracy but poor results, because the machine only needs to guess more. The classification, you can get a good-looking accuracy. Although there are good solutions already available[118], these are still important challenges for using AI with rare diseases.

Finally, when an AI model that can be used clinically is to be developed, proper verification settings must be ensured in the experimental verification of the model. Lack of sufficient verification can lead to untrustworthy models[119]. It is common that the training dataset and the test dataset are not extensive at the time of collection, thus resulting in poor results in practical applications.

FUTURE OF AI IN GASTROINTESTINAL RADIOLOGY

With advanced deep learning algorithm, computers can assist clinicians to make an accurate diagnostic decision by providing the right information. For difficulties in endoscopic and interventional procedure, however, information alone is of little help. Complete automation of a manual procedure must be assisted by both deep learning and robotics. For example, there have been marked advancements in robot-assisted endoscopy devices[120]. If these robots can be combined with an intelligent system that detect lesions via ultrasound[112], then it would have a potential to automatically take procure a biopsy sample from the lesion, or perform a surgical procedure, thus eliminating the difficulties of endoscopic and surgical technique.

The other factor that would augment the power of intelligent system is the development of radiological technology itself. The best example would be combination of radiology and endoscopic robotic capsule[121,122]. Recently, with the assistance of neural network, trajectory control and image visualization of endoscopic robotic capsules have been more automatic than they were previously[123]. In the future, if the size of ultrasound probe or other radiological device can be reduced to nanoscale, with an intelligent robotic capsule and intelligent ultrasound probe, fully automated detection and management of any lesion accessible by endoscopic capsules would be possible. Possible path to fully automatic diagnosis and intervention in gastroenterology by combining artificial intelligence with various technologies is shown in Figure 1.

Figure 1 Illustration of a possible path to automatic diagnostic and interventional system in gastroenterology.

The problems inherent to AI itself, that is, data acquisition and annotation, will also be solved by recent technical developments in deep learning models. The best sample would be using unsupervised learning or semi-supervised learning[16,18] to decrease or eliminate the need for radiologist annotation, making development of models faster. For research topics with large public database and well-developed models, such as abdominal CT, transfer learning with pre-trained model and included clinical data can also make training easier, more precise, and faster[124]. In addition to improvement of deep learning model itself, the advancement of advanced deep learning algorithm will enable in-vivo live visualization of lesion detection in endoscope[125], which will be a powerful, clinically applicable function. LeNet-5 architecture can be found in publication by Lecun et al[126].

However, areas with less data availability, such as EUS, cannot be advanced with AI technology alone. For developments of these areas, international collaboration for collection of multi-center image database and clinical data must be done to overcome data scarcity and facilitate precise training and evaluation of models. These multi-center database of image and clinical data will not only benefit model training, but also validation of previous models. Because multi-center data can be more unbiased than data from single source, validation or re-training by multi-center data may improve precision of models by eliminating sampling bias.

With future advancement in data science, deep learning algorithm and medical robotics, AI can play important role in gastrointestinal radiology in the future and may lead a medial revolution.

CONCLUSION

As demonstrated in the assistance of liver tumor diagnosis and cholangiography, AI has the potential to reduce radiology workload and improve diagnostic specificity, thus making radiologic diagnoses faster and more reliable. In some tasks like the detection of a malignant stricture, we can even hope for machine diagnosis to surpass human diagnosis, making fully automated diagnosis possible. Conversely, for fields where training data collection is more difficult, such as endoscopic ultrasound, training deep learning models would still be slow using today’s technology.

To overcome the problem of lack of technical advancement due to limited data in these areas, particularly in endoscopic procedure, two approaches maybe used. The first solution is to use algorithms that are designed to increase data availability in small medical dataset, such as generative adversarial network and transfer learning. The other suggestion is to build public, global endoscopic image library for model training. In conclusion, though a lot have to be done to make AI universally successful in gastrointestinal radiology, the researchers and developers actually already have the facility to deal with the difficult aspects of this task. Therefore, it is reasonable to expect more scientific advancements and clinical use of AI in the coming decade.