Augmented intelligence in “robotic” liver surgery: Integrating augmented reality and artificial intelligence for real-time navigation and margin precision

Abstract

“Robotic” liver surgery has revolutionized minimally invasive hepatectomy, offering superior dexterity and visualization, but achieving accurate intraoperative navigation and safe resection margins remains difficult, especially for deep or posterior hepatocellular carcinomas. The convergence of augmented reality (AR) and artificial intelligence (AI), collectively described as augmented intelligence, is emerging as a transformative approach. We review the current evidence and future prospects of AR- and AI-assisted “robotic” liver surgery, focusing on their role in real-time anatomical guidance and margin control. Recent developments allow for real-time overlay of 3D reconstructions of the liver and segmentation of tumors and vascular structures, facilitating dynamic visualization and intraoperative decision-making. However, surgical and anatomical challenges, for instance, organ deformation, registration errors, and the lack of standardized protocols, limit its widespread adoption. Augmented intelligence represents the next frontier in “robotic” liver surgery, offering a path toward safer, more personalized, and data-driven hepatectomies.

Key Words: Liver surgery; Robotic surgery; Artificial intelligence; Augmented intelligence; Hepatectomy

Core Tip: The integration of augmented reality and artificial intelligence into “robotic” liver surgery represents a transformative advancement in precision and decision-making. Surgeons can enhance real-time visualization of tumors, vasculature, and critical anatomy, reducing errors and improving surgical margins. However, challenges such as organ deformation and registration accuracy must be addressed for widespread adoption. As these innovative technologies evolve and standardization improves, augmented intelligence will not only boost the safety and precision of liver resections but also pave the way for personalized, data-driven surgical approaches that optimize robotic surgical outcomes for each patient.

INTRODUCTION

“Robotic” liver surgery, a cornerstone of modern minimally invasive hepatectomy, has undeniably changed the landscape of hepatobiliary surgery. The term “robotic surgery” warrants clarification, as current systems are not autonomous robots but rather platforms operated by surgeons with computational and artificial intelligence (AI) support. Accordingly, modern “robotic” liver surgery refers to procedures that are surgeon-guided and AI-assisted, providing enhanced precision while preserving the essential role of the surgeon in intraoperative decision-making and execution. With its enhanced precision, dexterity, and the ability to offer superior visualization, the robotic platform has significantly reduced patient morbidity and recovery time compared to traditional open surgery[1]. Yet, despite these advancements, challenges persist, especially when it comes to achieving accurate intraoperative navigation and maintaining safe resection margins. This issue is particularly critical when dealing with complex cases such as deep or posterior hepatocellular carcinomas, where precise visualization is essential for successful outcomes[1,2].

Enter augmented intelligence, an emerging synergy between augmented reality (AR) and AI, which promises to push the boundaries of what robotic surgery can achieve. In this new paradigm, real-time anatomical guidance and precise margin control may no longer be aspirational goals, but attainable realities. By integrating AR and AI, we may soon witness a transformative shift in the way surgeons approach liver resections, not only enhancing the accuracy of tumor visualization but also guiding decision-making in real-time[2]. The combination of AR and AI is transforming surgery by providing enhanced, data-driven decision-making and improving the real-time accuracy of surgical navigation[2]. In “robotic” liver surgery, augmented intelligence can offer significant advantages, particularly in the context of real-time anatomical guidance, tumor segmentation, and margin control[3].

THE RISE OF AUGMENTED INTELLIGENCE IN ROBOTIC SURGERY

The concept of augmented intelligence is built on the idea of amplifying human intelligence with the assistance of cutting-edge technologies like AR and AI. In “robotic” liver surgery, AR offers a way to overlay 3D reconstructions of the liver, vascular structures, and tumors onto the surgeon's field of view in real-time, often via heads-up displays or eyeglasses. This allows for dynamic, context-sensitive visualization that can adjust as the surgical scene changes, providing critical information that would otherwise remain hidden[4].

AI, on the other hand, brings the power of deep learning and pattern recognition to the table. Through sophisticated algorithms, AI can analyze preoperative imaging, segment tumors, predict tumor margins, and even identify critical vasculature that may not be immediately visible to the naked eye[4]. By continuously learning from vast datasets, AI can offer predictive insights and flag potential risks, providing surgeons with a comprehensive, data-driven view of the surgical environment[4,5]. Together, AR and AI offer an unprecedented level of intraoperative support. Surgeons are no longer relying solely on their visual and tactile senses to make decisions. With augmented intelligence, they are supported by a virtual layer of information that can enhance decision-making, reduce human error, and ultimately lead to safer, more precise liver resections[5,6].

AR FOR REAL-TIME VISUALIZATION

AR can offer a revolutionary advancement in liver surgery by providing the surgeon with a 3D view of the liver, its vasculature, and the location of tumors in real time. With AR, surgeons can visualize virtual overlays of the liver’s structure, superimposed onto the actual organ during surgery. This allows them to view deeper, difficult-to-reach structures such as posterior tumors or surrounding critical blood vessels[6]. AR systems typically use preoperative imaging [such as computed tomography (CT) or magnetic resonance imaging (MRI) scans] to create 3D models of the liver and its associated structures. These models are then registered onto the actual organ using advanced computer vision techniques[6,7].

AR has emerged as a transformative technology in the field of surgery, offering a range of benefits that significantly enhance both the precision and safety of surgical procedures. One of the most notable advantages of AR is its ability to provide enhanced visualization of anatomical structures, both superficial and deep, that would otherwise be difficult to see with the naked eye[8]. By overlaying virtual images onto the surgeon’s view of the patient, AR creates a more comprehensive and intuitive representation of the body, enabling the surgeon to visualize structures in three dimensions. This is particularly valuable in complex surgeries, such as those involving the brain, liver, or other organs where traditional imaging techniques may fall short[4,8,9].

Another significant benefit is the ability of AR systems to offer dynamic contextual overlays that adapt in real time to changes in the surgical field. As the surgeon makes adjustments or progresses through different stages of the procedure, the AR system can update and display critical information, such as the location of tumors, blood vessels, or nerves, as well as anatomical relationships that may shift during the operation. This continuous, real-time feedback ensures that the surgeon has access to the most current and relevant data, ultimately contributing to more informed decision-making during surgery[10,11].

Furthermore, AR has become a valuable tool for improving the precision of liver surgery, particularly in the accurate localization of tumors and other pathological lesions[11]. By projecting preoperative imaging (such as detailed CT or MRI datasets) directly onto the operative field, AR allows the surgeon to identify the exact position of a lesion with far greater confidence[10,11]. This level of precision supports safer parenchymal transection by reducing the risk of inadvertently injuring adjacent vital structures and helps ensure that resection margins remain oncologically sound. In situations where conventional imaging alone may not provide sufficient spatial clarity for complex operative planning, AR offers a more intuitive and anatomically faithful means of intraoperative navigation, enhancing the surgeon’s ability to orient within challenging anatomy[12,13].

AI FOR PREDICTIVE INSIGHTS AND DECISION SUPPORT

AI can complement AR by offering predictive insights based on large datasets of preoperative imaging, intraoperative sensor data, and historical clinical data[13]. AI models, particularly those based on deep learning algorithms, have demonstrated remarkable accuracy in tasks such as tumor segmentation, predicting surgical margins, and identifying vascular structures that are crucial to avoid during resection. AI’s ability to process large volumes of data enables it to offer real-time decision support during the surgery[11,13].

Automatically segment liver tumors from surrounding healthy tissue, providing clear, delineated tumor margins that may be difficult to identify visually during surgery. Predict optimal surgical margins, ensuring that the tumor is resected safely while minimizing the risk of recurrence. Detect changes in the surgical site, alerting the surgeon to potential complications such as bleeding or unexpected anatomical variations. The integration of AI with robotic systems can further enhance surgical precision[13-15]. For instance, AI can continuously update the virtual overlay, making adjustments based on real-time feedback from the robotic arms. This continuous loop of data collection and analysis can enhance surgical accuracy and speed, providing surgeons with a powerful tool to manage complex cases more effectively[16,17].

ADDRESSING INTRAOPERATIVE CHALLENGES

However, the path to widespread adoption of augmented intelligence in “robotic” liver surgery is not without obstacles. One of the most significant challenges is the issue of organ deformation. During surgery, the liver undergoes various degrees of deformation as it is manipulated, which can result in misalignment between preoperative imaging and the actual anatomy. This can lead to errors in navigation and a loss of accuracy in the overlay of virtual models onto the surgical site[15,17,18].

Another challenge is registration errors. For AR systems to function effectively, the virtual 3D models must align perfectly with the physical organ. This requires highly accurate registration techniques, which can be difficult to achieve in real-time, especially in the context of dynamic, constantly changing surgical conditions. Additionally, variations in individual anatomy and patient positioning further complicate the precision of these technologies[18,19].

Moreover, achieving accurate real-time registration of the liver’s anatomy remains difficult. Despite advancements in computer vision, current systems often struggle with the dynamic nature of the liver during surgery[19]. Addressing this challenge requires the development of advanced algorithms capable of compensating for these deformations and ensuring accurate overlay alignment throughout the surgical procedure[20]. Standardization also remains a key hurdle. While AI and AR technologies are rapidly evolving, there is no universally agreed-upon protocol for integrating these technologies into clinical practice[20,21]. Variations in hardware, software, and surgeon preferences may lead to inconsistencies in outcomes. Moreover, the training required to use these technologies effectively can be significant, further delaying their widespread adoption[21].

Despite the promising potential of augmented intelligence, several challenges need to be addressed before these technologies can be fully integrated into routine clinical practice[4,21]. One of the most significant obstacles is organ deformation. Liver tissue can change shape as it is manipulated during surgery, leading to discrepancies between preoperative imaging and actual surgical anatomy[11-13]. Intraoperative deformation due to manipulation, breathing motion, or blood flow changes can make it difficult for AR systems to maintain accurate overlays of the liver’s 3D model, potentially leading to registration errors and suboptimal navigation[13,14,19-21].

Registration errors, the misalignment between preoperative images and intraoperative reality, are another key challenge. Accurate alignment of the AR models to the patient’s anatomy is essential for the system to function correctly. Registration can be impacted by factors such as the patient's position on the operating table, organ movement, or changes in tissue properties during surgery. Furthermore, patient-specific variations in liver anatomy complicate the creation of universally accurate AR models, and achieving precise registration in real-time remains a significant hurdle[22].

Moreover, the lack of standardized protocols for implementing AR and AI in liver surgery is another barrier to widespread adoption. Different healthcare systems and robotic platforms may have varying hardware and software capabilities, creating inconsistencies in how these technologies are applied[6]. Furthermore, the training required to effectively use these advanced tools can be significant[6,8]. Surgeons must not only be proficient with robotic systems but also with interpreting AR overlays and integrating AI-driven insights into their decision-making process. This learning curve, combined with the need for specialized equipment, may delay the broad integration of augmented intelligence into clinical settings[6,8,21]. Table 1 outlines key studies on augmented intelligence in robotic liver surgery, comparing study design, technologies, and clinical implications.

Table 1 Key studies on augmented intelligence in “robotic” liver surgery.

Ref.	Procedures	Outcomes
Buchs et al[17]	Procedure: Robotic navigated atypical hepatic resection for hepatocellular carcinoma. Technology: Optical tracking of robotic instruments and the endoscopic camera. Real-time 3D model and virtual targeting superimposed on endoscopic video for precise tumor localization. Visualization of resection margins and relationship between tumor and instrument for safe surgery	Accurate tumor resection margins defined. 3D models used to identify vascular and biliary structures during parenchymal transection. Operative times: 240 minutes (case 1), 300 minutes (case 2). No intraoperative complications. Successful resections with safe margins and no adverse events
Pessaux et al[10]	Procedure: Robotic AR-assisted hepatic segmentectomy. 3D model generated from CT scans. AR superimposition of virtual model onto the operative field. Key tools: VR-RENDER® for model creation, VSP® for surgical planning, and a Panasonic MX 70 video mixer for real-time image registration	Precise identification of vascular structures. Short AR setup time (8 minutes). Minimal registration time (seconds). No need for hepatic pedicle clamping. Correct vascularization of remnant liver. Negative resection margins in all cases. Uneventful postoperative recovery without transfusion
Giannone et al[13]	Robotic liver resections (both benign and malignant lesions) using AR and other image-guided technologies. AR: Overlays 3D preoperative imaging (e.g., from CT or MRI scans) onto the live video feed from the robotic camera to guide the surgeon during the procedure. Preoperative 3D planning: Virtual models of the liver, tumors, and vascular structures are used to plan resection and avoid critical structures. Intraoperative guidance: Real-time superimposition of virtual models, such as tumor boundaries and vascular structures, during parenchymal transection. Robot-assisted system: The Da Vinci surgical system, integrated with imaging tools like AR, ultrasound, and indocyanine green fluorescence, to enhance precision	Enhanced tumor localization: AR allows for improved localization of tumors and resection margins during robotic liver resections. Surgical navigation: AR guides intraoperative procedures, such as parenchymal transection and vessel identification, improving precision and safety. Port placement guidance: Virtual 3D liver images help optimize port placement for robotic access, especially for challenging posterior segments. Increased accuracy in resection: AR allows surgeons to avoid critical structures and ensure clear resection margins, even in complex liver resections. Tactile feedback compensation: AR compensates for the lack of tactile feedback during robotic surgery, improving tumor identification and safe dissection. Visualize hidden lesions: Helps identify tumors not visible to traditional imaging or ultrasound. Preoperative planning and intraoperative real-time adjustment: 3D imaging aids in surgical planning, while real-time registration allows for dynamic adjustments based on the patient’s position and anatomy during surgery
Bijlstra et al[15]	Segmented liver, tumors, and vasculature from CT, MRI, and PET-CT scans. Used a deep-learning U-net for automatic liver segmentation (CT). Manual and semi-automatic methods for tumors and vasculature. 3D co-registration (Elastix) for combining segmented structures from different modalities. 3D models visualized using ParaView and ParaView Glance. Validation: NEMA-2012 phantom validation to assess segmentation accuracy. High interobserver agreement (dice similarity coefficient ≥ 0.87). 3DeliverS segmentation closely matched real dimensions	Surgical performance: 15 patients (13 with colorectal liver metastases, 2 with other conditions). No conversions to open surgery, no intraoperative incidents. 21 out of 22 lesions were malignant (CRLM). 19% of resections had positive margins (R1), with 3 intentional vascular resections. Measurement comparisons: Tumor diameters from CT/MRI similar to automated measurements in 3DeliverS software (P > 0.05). 3D measurements of tumor size slightly higher but not significantly different. Surgeon feedback (questionnaire): 93% of surgeons satisfied with 3D models. Most limiting factors: Missing portal and hepatic veins (33% and 20% of cases). Most beneficial aspects: Accurate tumor localization and proximity to vital structures
Gholizadeh et al[11]	Preoperative planning: Creation of 3D liver models from CT/MRI scans for surgical planning. Intraoperative navigation: AR visualization to enhance liver surgery, improving precision in identifying blood vessels, tumors, and other structures. 3D models overlaid onto the patient's body for real-time guidance during surgery	Successful visualization of blood vessels, tumors, and critical liver anatomy during surgery. Improved precision in navigating complex liver structures. AR demonstrated as both safe and effective in various liver surgeries, including both minimally invasive and open procedures. Facilitates precise navigation during complicated liver surgeries. Enhances surgical guidance, potentially reducing risks of injury to vital structures. AR technologies help improve preoperative planning and intraoperative decision-making. Shows potential for reducing postoperative morbidity and mortality, though more clinical trials are needed for confirmation
Oh et al[19]	4 right hemihepatectomies. 1 extended left hemihepatectomy. 1 left lateral sectionectomy. 4 segmentectomies. AR software overlays 3D digital models onto laparoscopic or robotic views for real-time surgical navigation	Registration alignment: Before mobilization (3.9 ± 1.1), after mobilization (4.1 ± 1.2). Helpfulness of AR software: Overall (4.2 ± 0.8). Locating structures: Blood vessels (4.2 ± 0.6), tumors (4.3 ± 0.7)

AR: Augmented reality; CT: Computed tomography; MRI: Magnetic resonance imaging; CRLM: Colorectal liver metastases; PET-CT: Positron emission tomography-computed tomography.

STANDARDIZATION AND PROTOCOLS FOR IMPLEMENTATION

Another challenge is the lack of standardized protocols for integrating AR and AI into “robotic” liver surgery. Different hospitals, research centers, and robotic platforms may use different hardware, software, and imaging techniques, leading to inconsistencies in how augmented intelligence is applied. Additionally, there is no universally agreed-upon workflow for surgeons to incorporate these technologies into their routine practice. This lack of standardization can make it difficult to assess the true clinical benefits of augmented intelligence and may delay its adoption[9,21,22].

TRAINING AND FAMILIARIZATION WITH NEW TECHNOLOGIES

The learning curve for surgeons to effectively use AR and AI tools is another hurdle. Surgeons are already highly trained in conventional methods of liver surgery and must now learn how to integrate these advanced technologies into their workflow[11]. This training requires not only technical skills in handling robotic systems but also proficiency in interpreting AR overlays and understanding AI-driven decision support in real-time. As a result, there is a need for targeted educational initiatives and simulation-based training programs to help surgeons become proficient in these technologies[11,15,18].

The integration of AI into surgery requires access to vast amounts of data from preoperative imaging, intraoperative sensors, and electronic health records. This raises concerns regarding data privacy, especially when dealing with sensitive patient information. Additionally, there needs to be a seamless integration of AI systems with existing clinical workflows and medical records systems, ensuring that AI models can access and analyze data without disrupting clinical operations[19,20].

FUTURE PROSPECTS: PERSONALIZED, DATA-DRIVEN LIVER SURGERY

Despite these challenges, the future of “robotic” liver surgery lies in the integration of augmented intelligence to provide a more personalized, data-driven approach. By leveraging data from multiple sources, including preoperative imaging, intraoperative sensor data, and even genomic and clinical data, AI can tailor the surgical approach to the individual patient. Personalized surgical plans based on a patient’s unique anatomy, tumor characteristics, and surgical history could significantly reduce complications and improve outcomes[3,4,10].

Additionally, the ability to predict complications in real-time and adjust the surgical approach accordingly will be a major benefit of augmented intelligence. As AI algorithms learn from a broader pool of data over time, they will become increasingly adept at forecasting surgical risks and recommending interventions that enhance patient safety[10,17]. The integration of haptic feedback with AR and AI could further enhance the surgeon’s capabilities. Haptic feedback, integrated into robotic systems, could provide tactile cues that correspond to the surgeon’s visual field, making it easier to navigate complex anatomical structures or identify subtle changes in tissue during resection. This combination of visual, tactile, and cognitive inputs could lead to an even more intuitive and efficient surgical workflow[15,17,23-25].

PERSONALIZING “ROBOTIC” LIVER SURGERY WITH DATA

Despite these challenges, the future of augmented intelligence in “robotic” liver surgery holds great promise. The ability to integrate vast amounts of data, from pre-operative imaging to real-time intraoperative sensor data, can create highly personalized and tailored surgical plans[23]. By factoring in each patient’s unique anatomical features, tumor characteristics, and even their response to previous treatments, augmented intelligence could pave the way for more individualized, precision-driven surgeries[17,22,23,25].

The potential benefits extend beyond mere precision. Augmented intelligence may also improve the reproducibility of surgical outcomes. With AI continuously learning from large datasets of liver surgeries, it can not only identify patterns and potential pitfalls but also suggest optimal strategies for various clinical scenarios[23]. Over time, this could lead to standardized best practices, reducing variability and enhancing the overall quality of care[23,25].

Moreover, as these technologies evolve, we may witness a convergence of AR and AI with other cutting-edge tools, such as robotic systems with haptic feedback, which could further enhance the surgeon’s ability to navigate complex hepatic anatomy[25]. The integration of machine learning algorithms with robotic platforms could allow for continuous feedback, enabling surgeons to adapt in real-time to any unforeseen complications or changes during the operation[17,19,21].

CONCLUSION

The integration of AR and AI into “robotic” liver surgery marks a pivotal moment in the evolution of hepatobiliary surgery. While the technology is still in its infancy, the potential to enhance surgical precision, improve margin control, and reduce human error is undeniable. As challenges related to organ deformation, registration accuracy, and standardization are addressed, augmented intelligence could become a routine part of surgical practice, offering a new era of safer, more efficient, and personalized liver resections.

From the perspective of a hepatobiliary surgeon, the integration of augmented intelligence into “robotic” liver surgery represents a transformative change in both surgical strategy and surgical knowledge. Traditionally, successful liver resection has relied on the surgeon's ability to synthesize static images with real-time intraoperative findings, often under conditions of limited visibility and limited anatomical access. AR and AI offer a means to bridge this gap, enabling dynamic, context-sensitive visualization of critical structures and providing data-driven information during important decision-making. In my experience, the integration of 3D reconstructions and real-time overlay has improved the accuracy of tumor localization and enhanced surgical confidence in anatomically complex cases, particularly those involving deep parenchymal lesions or tumors adjacent to vital vascular structures. These tools do not diminish the importance of surgical judgment, but rather enhance it by supplementing human perception with computational analysis that improves safety and accuracy.

Surgeons who embrace these technologies will not only be at the forefront of innovation but will also provide their patients with the best possible outcomes. As we look to the future, augmented intelligence represents not just a tool for improving current practices, but a transformative force that will shape the next generation of robotic liver surgeries. The convergence of AR and AI offers a promising path forward, one that could redefine the boundaries of what is possible in hepatobiliary surgery.

Despite this promise, I firmly believe that the widespread implementation of augmented intelligence will depend on a robust expansion of high-quality surgical data and rigorous clinical research. Existing datasets are often fragmented, limited in size, or derived from heterogeneous imaging protocols and operative environments. To meaningfully improve AI model accuracy, expand applicability across diverse patient populations, and reduce algorithmic bias, the field urgently needs large-scale, multicenter repositories that integrate preoperative imaging, intraoperative video and sensor data, postoperative outcomes, and long-term oncologic results. Moreover, prospective clinical trials, carefully designed to evaluate safety, reproducibility, and cost-effectiveness, are essential for validating whether these technologies truly enhance surgical performance beyond what expert surgeons can achieve unaided. Without this level of methodological rigor, augmented intelligence risks remaining a promising but unproven adjunct rather than an evidence-based standard of care. In my view, the next stage of progress will require interdisciplinary collaboration among surgeons, engineers, data scientists, and regulatory bodies to develop standardized workflows, interoperable platforms, and ethical frameworks that ensure safe, responsible, and equitable implementation. Advanced simulation-based training programs must also be established to ensure that surgeons can safely learn and integrate these tools without compromising patient outcomes.