Clinical use of augmented reality, mixed reality, three-dimensional-navigation and artificial intelligence in liver surgery

Table 1 Augmented and mixed reality in open liver surgery

Ref.	No of procedures	Technique	Key outcomes
Onda et al[11], 2013	2 liver resections	Open stereo-scope, AR created on a passive -polarizing 3D display	Open scope technique feasible, 10 hr pre-op image preparation, 1 h intraoperative setup, 1-2 min for registration process
Okamoto et al[12], 2013	2 HPB procedures	Video see-through display	Position of virtual 3D model and organ image closely corresponded, registration error 5 mm
Ntourakis et al[13], 2016	3 patients with 4 disappeared CRLM	Open stereo-scope, AR created on video screen, registration performed by an additional computer technician	AR helped to detect disappeared all metastases, R0, planned security margin 1 cm, registration time within 6 min
Tang et al, 2017[14]	1 patient	AR created on a tablet pc as see-trough display	Feasible, improved vision compared to video based AR system
Yasuda et al[15], 2018	7 patients including minor and major liver resections	Open scope technique combined with AR created on a tablet pc with infrared sensor	Tablet pc method feasible, registration error 1-11 mm
Saito et al[18], 2020	2 HPB procedures	3D hologram on head mounted display	Feasible, orientation improved, multiple surgeons used the technique at the same time, hologram reduced task load

CRLM: Colorectal liver metastasis; HPB: Hepato-biliary; AR: Augmented reality; 3D: Three-dimensional; R0: R0 Resection.

Table 2 Augmented and mixed reality in minimally invasive liver surgery

Ref.	No of procedures	Technique	Key outcomes
Volonté et al[19], 2011	4 procedures	Projection of the virtual 3D model on the body surface	Anatomical orientation and trocar placement improved
López-Mir et al[20], 2013	12 procedures	Projection of the virtual 3D model on the body surface	lower deviation between planned and actual trocar positions using AR
Pessaux et al[25], 2015	2 robotic liver resections	Virtual 3D model superimposed on console display, registration performed manually by a computer scientist	AR and registration process feasible, time to create AR 8 min
Schneider et al[22], 2020	18 laparoscopic liver resections	Passive polarizing 3D laparoscope, optical tracking of the laparoscope, semi-automatic registration	semiautomatic registration an image fusion achieved in 16/18 manual registration vs semiautomatic accuracy 11 mm vs 14 mm

AR: Augmented reality, 3D: Three-dimensional.

Table 3 Augmented and mixed reality for 3D Navigation

Ref.	Number of procedures	Technique	Key outcomes
Beller et al[29], 2007	33 open liver resections	Stereotactic optical navigation system, combined with a virtual 3D model and ultrasound, dissection device tracked and navigated on ultrasound image	Navigation successful in 32/33 cases, difference between projected and actual vascular dissection lever 6mm, R0 resection in 30 cases
Peterhans et al[10], 2011	9 open liver resections	Stereotactic navigation system, combined with a virtual 3D model and ultrasound, landmark acquisition on the liver surface, dissection device tracked and navigated on the virtual 3D model	Navigation successful in all cases, median accuracy 6.3 mm
Banz et al[32], 2016	65 open liver resections	Stereotactic optical navigation system, combined with a virtual 3D model and ultrasound, dissection device tracked and navigated on the virtual 3D model, landmark acquisition with ultrasound possible	Combination of 3 d navigated resection and thermal ablation in 16 patients, accuracy optimized to 4.5 ± 3.6 mm
Tinguely et al[35], 2017	54 laparoscopic image guided microwave ablation	Laparoscopic stereotactic navigation system, combined with a virtual 3D model, landmark acquisition on the liver surface, ablation device tracked and navigated on the virtual 3D model, standard 2D laparoscopic display	Registration time 4:38 min, accuracy 8.1 ± 2.8 mm, early local recurrence rate 32%
Aoki et al[37], 2021	27 laparoscopic liver lesions	virtual real-time CT-guided volume navigation, electromagnetic tracking of the surgical instruments displayedon the preoperatively acquired CT images	Registration time < 2 min, registration error 12 mm, histologic resection margin 9 mm
Prevost et al[26], 2020	10 laparoscopic liver resections	stereotactic augmented reality navigation, virtual 3D liver model superimposed on the real liver with a 3D laparoscopic system, instruments tracked	Registration time 8:50 min, registration error 9.2 mm, facilitates to find disappeared liver lesions

AR: Augmented reality; 3D: Three-dimensional.