Large language models and large concept models in radiology: Present challenges, future directions, and critical perspectives

Abstract

Large language models (LLMs) have emerged as transformative tools in radiology artificial intelligence (AI), offering significant capabilities in areas such as image report generation, clinical decision support, and workflow optimization. The first part of this manuscript presents a comprehensive overview of the current state of LLM applications in radiology, including their historical evolution, technical foundations, and practical uses. Despite notable advances, inherent architectural constraints, such as token-level sequential processing, limit their ability to perform deep abstract reasoning and holistic contextual understanding, which are critical for fine-grained diagnostic interpretation. We provide a critical perspective on current LLMs and discuss key challenges, including model reliability, bias, and explainability, highlighting the pressing need for novel approaches to advance radiology AI. Large concept models (LCMs) represent a nascent and promising paradigm in radiology AI, designed to transcend the limitations of token-level processing by utilizing higher-order conceptual representations and multimodal data integration. The second part of this manuscript introduces the foundational principles and theoretical framework of LCMs, highlighting their potential to facilitate enhanced semantic reasoning, long-range context synthesis, and improved clinical decision-making. Critically, the core of this section is the proposal of a novel theoretical framework for LCMs, formalized and extended from our group’s foundational concept-based models - the world’s earliest articulation of this paradigm for medical AI. This conceptual shift has since been externally validated and propelled by the recent publication of the LCM architectural proposal by Meta AI, providing a large-scale engineering blueprint for the future development of this technology. We also outline future research directions and the transformative implications of this emerging AI paradigm for radiologic practice, aiming to provide a blueprint for advancing toward human-like conceptual understanding in AI. While challenges persist, we are at the very beginning of a new era, and it is not unreasonable to hope that future advancements will overcome these hurdles, pushing the boundaries of AI in Radiology, far beyond even the most state-of-the-art models of today.

Keywords: Radiology artificial intelligence; Large language models; Large concept models; Medical imaging artificial intelligence; Artificial intelligence in healthcare; Multimodal artificial intelligence models; Explainable artificial intelligence; Artificial intelligence model limitations and challenges; Natural language processing in radiology; Conceptual reasoning in artificial intelligence

Core Tip: Current capabilities, applications, limitations of large language models (LLMs) in radiology artificial intelligence (AI). LLMs transformed radiology AI, improved textual-analysis, workflow automation, clinical decision support. Challenges: Limited reasoning depth, inaccuracies. Transformative role of LLMs in radiology, their architectural foundations, clinical utility are discussed. LLM limitations, like token-level processing, hallucinations, and challenges in clinical adoption. Exploring new paradigm large concept models, having conceptual reasoning and multimodal integration to enhance clinical accuracy and reliability. Ethical, regulatory, and explainability considerations for AI tools in healthcare also discussed and a balanced and forward-looking view on AI’s role in radiology, covering both current innovations and anticipated advances through large concept models.