Augmented intelligence in “robotic” liver surgery: Integrating augmented reality and artificial intelligence for real-time navigation and margin precision

Table 1 Key studies on augmented intelligence in “robotic” liver surgery

Ref.	Procedures	Outcomes
Buchs et al[17]	Procedure: Robotic navigated atypical hepatic resection for hepatocellular carcinoma. Technology: Optical tracking of robotic instruments and the endoscopic camera. Real-time 3D model and virtual targeting superimposed on endoscopic video for precise tumor localization. Visualization of resection margins and relationship between tumor and instrument for safe surgery	Accurate tumor resection margins defined. 3D models used to identify vascular and biliary structures during parenchymal transection. Operative times: 240 minutes (case 1), 300 minutes (case 2). No intraoperative complications. Successful resections with safe margins and no adverse events
Pessaux et al[10]	Procedure: Robotic AR-assisted hepatic segmentectomy. 3D model generated from CT scans. AR superimposition of virtual model onto the operative field. Key tools: VR-RENDER® for model creation, VSP® for surgical planning, and a Panasonic MX 70 video mixer for real-time image registration	Precise identification of vascular structures. Short AR setup time (8 minutes). Minimal registration time (seconds). No need for hepatic pedicle clamping. Correct vascularization of remnant liver. Negative resection margins in all cases. Uneventful postoperative recovery without transfusion
Giannone et al[13]	Robotic liver resections (both benign and malignant lesions) using AR and other image-guided technologies. AR: Overlays 3D preoperative imaging (e.g., from CT or MRI scans) onto the live video feed from the robotic camera to guide the surgeon during the procedure. Preoperative 3D planning: Virtual models of the liver, tumors, and vascular structures are used to plan resection and avoid critical structures. Intraoperative guidance: Real-time superimposition of virtual models, such as tumor boundaries and vascular structures, during parenchymal transection. Robot-assisted system: The Da Vinci surgical system, integrated with imaging tools like AR, ultrasound, and indocyanine green fluorescence, to enhance precision	Enhanced tumor localization: AR allows for improved localization of tumors and resection margins during robotic liver resections. Surgical navigation: AR guides intraoperative procedures, such as parenchymal transection and vessel identification, improving precision and safety. Port placement guidance: Virtual 3D liver images help optimize port placement for robotic access, especially for challenging posterior segments. Increased accuracy in resection: AR allows surgeons to avoid critical structures and ensure clear resection margins, even in complex liver resections. Tactile feedback compensation: AR compensates for the lack of tactile feedback during robotic surgery, improving tumor identification and safe dissection. Visualize hidden lesions: Helps identify tumors not visible to traditional imaging or ultrasound. Preoperative planning and intraoperative real-time adjustment: 3D imaging aids in surgical planning, while real-time registration allows for dynamic adjustments based on the patient’s position and anatomy during surgery
Bijlstra et al[15]	Segmented liver, tumors, and vasculature from CT, MRI, and PET-CT scans. Used a deep-learning U-net for automatic liver segmentation (CT). Manual and semi-automatic methods for tumors and vasculature. 3D co-registration (Elastix) for combining segmented structures from different modalities. 3D models visualized using ParaView and ParaView Glance. Validation: NEMA-2012 phantom validation to assess segmentation accuracy. High interobserver agreement (dice similarity coefficient ≥ 0.87). 3DeliverS segmentation closely matched real dimensions	Surgical performance: 15 patients (13 with colorectal liver metastases, 2 with other conditions). No conversions to open surgery, no intraoperative incidents. 21 out of 22 lesions were malignant (CRLM). 19% of resections had positive margins (R1), with 3 intentional vascular resections. Measurement comparisons: Tumor diameters from CT/MRI similar to automated measurements in 3DeliverS software (P > 0.05). 3D measurements of tumor size slightly higher but not significantly different. Surgeon feedback (questionnaire): 93% of surgeons satisfied with 3D models. Most limiting factors: Missing portal and hepatic veins (33% and 20% of cases). Most beneficial aspects: Accurate tumor localization and proximity to vital structures
Gholizadeh et al[11]	Preoperative planning: Creation of 3D liver models from CT/MRI scans for surgical planning. Intraoperative navigation: AR visualization to enhance liver surgery, improving precision in identifying blood vessels, tumors, and other structures. 3D models overlaid onto the patient's body for real-time guidance during surgery	Successful visualization of blood vessels, tumors, and critical liver anatomy during surgery. Improved precision in navigating complex liver structures. AR demonstrated as both safe and effective in various liver surgeries, including both minimally invasive and open procedures. Facilitates precise navigation during complicated liver surgeries. Enhances surgical guidance, potentially reducing risks of injury to vital structures. AR technologies help improve preoperative planning and intraoperative decision-making. Shows potential for reducing postoperative morbidity and mortality, though more clinical trials are needed for confirmation
Oh et al[19]	4 right hemihepatectomies. 1 extended left hemihepatectomy. 1 left lateral sectionectomy. 4 segmentectomies. AR software overlays 3D digital models onto laparoscopic or robotic views for real-time surgical navigation	Registration alignment: Before mobilization (3.9 ± 1.1), after mobilization (4.1 ± 1.2). Helpfulness of AR software: Overall (4.2 ± 0.8). Locating structures: Blood vessels (4.2 ± 0.6), tumors (4.3 ± 0.7)

AR: Augmented reality; CT: Computed tomography; MRI: Magnetic resonance imaging; CRLM: Colorectal liver metastases; PET-CT: Positron emission tomography-computed tomography.