Foundation models: Insights and implications for gastrointestinal cancer

Abstract

Gastrointestinal (GI) cancers represent a major global health concern due to their high incidence and mortality rates. Foundation models (FMs), also referred to as large models, represent a novel class of artificial intelligence technologies that have demonstrated considerable potential in addressing these challenges. These models encompass large language models (LLMs), vision FMs (VFMs), and multimodal LLMs (MLLMs), all of which utilize transformer architectures and self-supervised pre-training on extensive unlabeled datasets to achieve robust cross-domain generalization. This review delineates the principal applications of these models: LLMs facilitate the structuring of clinical narratives, extraction of insights from medical records, and enhancement of physician-patient communication; VFMs are employed in the analysis of endoscopic, radiological, and pathological images for lesion detection and staging; MLLMs integrate heterogeneous data modalities, including imaging, textual information, and genomic data, to support diagnostic processes, treatment prediction, and prognostic evaluation. Despite these promising developments, several challenges remain, such as the need for data standardization, limited diversity within training datasets, substantial computational resource requirements, and ethical-legal concerns. In conclusion, FMs exhibit significant potential to advance research and clinical management of GI cancers. Future research efforts should prioritize the refinement of these models, promote international collaborations, and adopt interdisciplinary approaches. Such a comprehensive strategy is essential to fully harness the capabilities of FMs, driving substantial progress in the fight against GI malignancies.

Keywords: Foundation models; Gastrointestinal cancers; Large language models; Vision foundation models; Multimodal large language models

Core Tip: This review synthesizes applications of foundation models in gastrointestinal cancer, from clinical text structuring and image analysis to multimodal data integration. Despite current knowledge gaps and challenges like data standardization, it highlights foundation models’ transformative potential, urging refined models and collaborations to advance gastrointestinal cancer research.