Comparing the reliability of three software programmes for manual and (semi-)automatic volumetry in healthy livers

Abstract

BACKGROUND

Multiple software programs are available for manual or (semi-)automated liver segmentation. Although manual segmentation has traditionally been the standard, it is time-intensive and susceptible to inter-observer variability. Conversely, (semi-)automated approaches have been shown to reduce segmentation time, but the availability and use of different software tools can introduce variability in liver volumetry outcomes.

AIM

To evaluate the reliability of three segmentation software programs (Synapse 3D (Fujifilm), OsiriX^® (Pixmea), and Syngo.via (Siemens Healthineers)) for liver volumetry.

METHODS

Patients (≥ 18 years) with abdominal computed tomography images without liver pathology were included. Total liver volume (TLV) and segment II/III volume – a surrogate for the future liver remnant (FLR) – were measured. Two blinded independent researchers performed volumetric assessments. The intraclass correlation coefficient (ICC) was calculated for interobserver variability for the first 30 scans between both researchers. This was also performed for intra-observer variability between the software programs in 200 randomly selected patients (52.5% male; mean age: 63.5 years).

RESULTS

Mean TLV’s reported by Synapse 3D, Syngo.via and OsiriX^® were 1705 ± 462 mL, 1672 ± 463 mL, and 1627 ± 449 mL, respectively. Mean segment II/III volumes reported were 304 ± 115 mL, 282 ± 109 mL, and 277 ± 108 mL, respectively. The highest interobserver agreement (n = 30) was for TLV with Synapse 3D (ICC: 0.999; 95%CI: 0.998-0.999), and highest intraobserver ICC was between Synapse 3D and Syngo.via for TLV (ICC: 0.997; 95%CI: 0.985-0.999). Mean standardized FLR for Synapse 3D, Syngo.via and OsiriX^® were 18.6%, 17.2%, and 16.9%, respectively.

CONCLUSION

Although absolute volumes varied slightly, the standardized FLR remained comparable across the software programs, indicating that all three are suitable for reliable computed tomography liver volumetry.

Key Words: Liver malignancy; Liver volumetry software; (Semi)automated segmentation; Manual segmentation; Imaging-based volumetry

Core Tip: This study compared three software programs (semi-automatic and manual) for computed tomography-based liver volumetry in 200 patients. Total liver volume and segment II/III volume – used as a surrogate for the future liver remnant – showed excellent interobserver agreement and strong agreement between all three software programs. Although Synapse 3D, Syngo.via, and OsiriX^® produced small systematic differences in absolute volume estimates, the standardized future liver remnant remained comparable. These findings indicate that all three programs can provide reliable liver volumetry, while highlighting that absolute measurements should be interpreted in relation to the software used.

INTRODUCTION

Liver volumetry, often used to calculate the future liver remnant (FLR), is frequently performed by radiologists, surgeons or other treating physicians using segmentation software programs on computed tomography (CT) images[1-3]. Adequate liver segmentation must account for the complex and variable shape of the liver, presence of tumors, and anatomical variations[4]. In the past 15 years, software programs have improved substantially. Calculation of liver volumes can either be performed using manual or (semi-)automated segmentation software programs[2,5-7]. Manual segmentation is more time intensive and prone to interobserver variability[2]. Six studies have compared manual and (semi-)automatic software programs[1,8-12]. Both methods lead to similar liver volumes, but reduced segmentation time has been demonstrated when an (semi-)automated approach is used[9-13]. Most studies evaluated a single segmentation software tool or compared one (semi-)automated software program to a manual approach in patients undergoing surgery. As manual segmentation in these cases often requires subjective adjustments, this approach is prone to high inter-observer variability, which may confound comparisons of software performances across studies[13]. Additionally, studies comparing multiple (semi-)automatic software tools with manual segmentation in large sample sizes remain limited. Therefore, the aim of this study was, first, to evaluate the performance of three segmentation software programs for liver volumetry and segmentation and, second, to assess variability between observers and among the programs.

MATERIALS AND METHODS

Study design and patient characteristics

Patients ≥ 18 years who underwent contrast-enhanced abdominal CT imaging with full coverage of the liver at Maastricht University Medical Centre+ between January 2023 and May 2023 were retrospectively included.

Exclusion criteria: (1) History of liver pathology or surgery; (2) Free abdominal fluid or air; (3) History of oncological diseases; and (4) Poor imaging quality (e.g. metal or movement artefacts). Additionally, patients were excluded when baseline characteristics or imaging data were missing. A total of 200 patients were randomly selected. The data were stored in an online secure database (Castor EDC, v2020.2.20). All imaging data were pseudonymized. The local medical ethics committee approved the study protocol and waived the need for obtaining informed consent (No. 2023-0185).

CT liver volumetry

Portal venous series with 3 mm slice thickness from contrast-enhanced CT scans were used for liver volumetry. More information regarding the scanning protocols is shown in Supplementary Table 1. The segmentation software programs compared were Syngo.via Client 10 (Siemens Healthcare, Munich, Germany), Synapse 3D (Fujifilm Corporation, Tokyo, Japan), and OsiriX^® (Pixmea, Switzerland) for Apple Mac OS. Synapse 3D and Syngo.via have (semi-)automated liver segmentation analysis functions with automated liver volume calculation. In OsiriX^®, liver segmentation was performed with manual delineation of the liver. Segmentation was performed for total liver volume (TLV) and Couinaud liver segment II/III (SII/III) in milliliters (mL)[14]. SII/III volume was considered as a surrogate for the FLR. To minimize systematic differences in liver volumetry, SII/III volumes were selected as their anatomy allows for reliable delineation. TLV segmentation in Syngo.via and Synapse 3D was checked and corrected manually when needed. During total liver segmentation, intrahepatic vascular and hepatic biliary structures were included on all slices across all three software programs. Anatomical structures excluded from segmentation were the inferior vena cava and the gallbladder. Prior to liver volumetry, observer 2 (RK, with > 3 years’ experience) instructed observer 1 (LvdV, no prior experience, trained on the software immediately beforehand) on how to use all three software programs.

For SII/III, volume calculation/delineation was performed manually by the two observers. The first 30 CT scans were segmented independently by both observers on all three software programs. During segmentation, both observers were blinded to each other’s volumetry results and patient characteristics. Liver volumetry time per patient was measured in seconds using a stopwatch by both observers. Time started from the first contour drawing with OsiriX^® or start of semi-automated segmentation of the whole liver in Syngo.via or Synapse 3D, and ended once segmentation of both TLV and SII/III was finalized. Figure 1 shows the segmentation process per software program.

Figure 1 Segmentation example of all three software programs. A: Synapse 3D; B: Syngo.via; C: Osirix^®.

Statistical analysis

Demographic and clinical data such as age, sex, weight, and length were collected from electronic patient files. Categorical data were presented as n (%) and continuous data as mean and SD or median and interquartile range, depending on the data distribution.

Comparison between observers (interobserver variability)

For testing the interobserver variability between observer 1 and 2, the intraclass correlation coefficient (ICC) was calculated after the first 30 scans, as described by Koo and Li[15]. Categorization for amount of agreement in ICC was as follows: (1) Poor (ICC < 0.500); (2) Moderate (ICC: 0.500-0.750); (3) Good (ICC: 0.750-0.900); and (4) Excellent (ICC > 0.900) agreement. The two-way-random-effects model for consistency was used to calculate the interobserver variability for the difference in TLV and SII/III between the observers. If the ICC was considered at least good between both observers, the remaining 170 scans for segmentation were divided between the two observers. FLR% = (SII/III volume)/TLV × 100 was calculated to determine whether systematic overestimation or underestimation occurred within each program, thereby identifying potential software-related bias in the volumetric results.

Additionally, standardized FLR (sFLR) was calculated as sFLR= (SII/III volume)/(-794 + 1267 × body surface area) × 100, according to the method presented by Ribero et al[16], in which liver volume correlates to the body surface area. In this manner, a clinically relevant metric was provided to compare the outcomes across the three software programs. Differences in absolute TLV, SII/III volumes, liver volumetry time, sFLR and FLR% between the two observers were tested using the Wilcoxon signed rank test[17].

Comparison among the software programs (intraobserver variability)

For testing the intraobserver variability among the three software programs, a two-way mixed-effect ICC for absolute agreement was used, as described by Koo and Li[15], with similar agreement categorization as previously described. The ICC between the software programs was calculated for all (n = 200) scans. Bland-Altman plots with 95% limits of agreement were used to compare OsiriX^®, Syngo.via and Synapse 3D[16]. For TLV and SII/III volumes, the difference was plotted against the mean, and the 95% limit of agreement was provided. The sFLR and FLR% of all three software programs were also calculated for all scans.

The paired-samples t-test was used to analyze significant differences in mean volume, segmentation time, FLR%, and sFLR between the software programs for all scans. Statistical analyses and graphical plots were performed using R (version 4.2.1). Results were considered statistically significant when P ≤ 0.05.

RESULTS

Baseline characteristics

Of the 4097 identified CT abdominal scans performed between January 2023 and May 2023, 1199 potential patients with a portal venous phase CT abdominal scan were included. Of these patients, 200 that satisfied all inclusion criteria were randomly selected. In this cohort, 105 (52.5%) of the patients were male. The mean age was 63.5 ± 15.7 years, and the mean body mass index was 26.7 ± 5.2 kg/m².

Comparison between observers

The ICCs for TLV and SII/SIII volumes for the first 30 abdominal CT scans between observer 1 and observer 2 were in excellent agreement for all three software programs (Table 1). The highest agreement was found for Synapse 3D TLV (ICC = 0.999; 95%CI: 0.998-0.999). The lowest agreement between the observers was observed for SII/III volumetry with Syngo.via (ICC = 0.958, 95%CI: 0.912-0.980).

Table 1 The intraclass correlation coefficient between the two observers in each liver volume program.

Item	n	ICC	95%CI
Syngo.via TLV	30	0.999	0.997-0.999
Syngo.via SII/III	30	0.958	0.912-0.980
OsiriX® TLV	30	0.996	0.992-0.998
OsiriX® SII/III	30	0.977	0.952-0.989
Synapse 3D TLV	30	0.999	0.998-0.999
Synapse 3D SII/III	30	0.975	0.948-0.988

The differences in volumes for total liver volume and segment II/III between both observers are shown in Table 2. For all three software programs, statistical differences were found for total liver volume and segment II/III volumes between the two observers (P value < 0.020). ICC: Intraclass correlation coefficient; TLV: Total liver volume; SII/III: Segment II/III.

Table 2 Liver segmentation volumes from the observers and n = 30 scans, median (interquartile range).

Item	Observer 1	Observer 2	P value	Median percent difference between observers
OsiriX®
TLV	1523 (1436-1846)	1513 (1422-1812)	0.019	1.4 (-0.7 to 2.5)
SII/III	289 (228-349)	287 (222-347)	0.020	-4.4 (-8.4 to 0.8)
Time in secs	480 (420-600)	660 (540-720)	0.013	-
Syngo.via
TLV	1524 (1392-1873)	1544 (1417-1894)	< 0.001	-0.9 (-2.2 to 0.0)
SII/III	253 (215-290)	282 (215-329)	0.002	-6.4 (-13.9 to 1.0)
Time in secs	390 (255-420)	180 (180-240)	< 0.001	-
Synapse 3D
TLV	1525 (1428-1945)	1530 (1463-1955)	0.006	-0.13 (-1.7 to 1.3)
SII/III	284 (236-350)	316 (263-367)	< 0.001	-10.4 (-13.8 to 6.1)
Time in secs	240 (180-300)	180 (120-180)	0.003	-

TLV: Total liver volume; SII/III: Segment II/III.

Regarding median FLR% and median sFLR, significant differences were found between the observers for all three software programs. The FLR% of observer 1 and observer 2 were 15.7 (13.2-18.7) vs 17.6 (13.9-19.6) (P = 0.008) for Syngo.via, 18.3 (15.6-20.6) vs 20.1 (17.8-22.3) (P < 0.001) for Synapse 3D, and 18.3 (14.5-20.8) vs 18.4 (15.4-21.5) (P = 0.015) for OsiriX^®, respectively. The sFLR of observer 1 and observer 2 were 14.7 (12.3-19.6) vs 16.7 (12.9-21.5) (P = 0.006) for Syngo.via, 17.9 (12.6-22.6) vs 19.2 (3.5-23.1) (P = 0.004) for Synapse 3D, and 17.2 (12.8-21.2) vs 18.5 (13.0-22.0) (P = 0.010) for OsiriX^®, respectively.

In regard to volumetry time, longer segmentation duration was seen for observer 1 for Syngo.via [median: 390 seconds (255-420) vs 180 seconds (180-240); P < 0.001] and Synapse 3D [median: 240 seconds (180-300) vs 180 seconds (120-180); P = 0.003] compared to observer 2 (Table 2).

Comparison of the volumetry software programs

For all scans, the calculated ICC between the software programs for TLV and SII/III volumes are shown in Table 3. For TLV, excellent agreement was observed between all software programs ranging from the lowest ICC between Synapse 3D and OsiriX^® (ICC = 0.989; 95%CI: 0.933-0.996; P < 0.001), and the highest ICC between Synapse 3D and Syngo.via (ICC = 0.992; 95%CI: 0.974-0.996; P < 0.001). For SII/III volumetry, the lowest ICC was observed between Synapse 3D and OsiriX^® (ICC = 0.939; 95%CI: 0.715-0.975; P < 0.001), and the highest ICC was seen between Synapse 3D and Syngo.via, showing an ICC of 0.944 (95%CI: 0.927-0.958; P < 0.001).

Table 3 The intraclass correlation coefficient between the three liver volumetry software programs for all scans (n = 200).

Item	ICC	95%CI	P value
Syngo.via vs OsiriX® TLV	0.989	0.933-0.996	< 0.001
Synapse 3D vs OsiriX® TLV	0.990	0.655-0.997	< 0.001
Synapse 3D vs Syngo.via TLV	0.997	0.985-0.999	< 0.001
Syngo.via vs OsiriX® SII/III	0.944	0.927-0.958	< 0.001
Synapse 3D vs OsiriX® SII/III	0.939	0.715-0.975	< 0.001
Synapse 3D vs Syngo.via SII/III	0.937	0.834-0.968	< 0.001

ICC: Intraclass correlation coefficient; TLV: Total liver volume; SII/III: Segment II/III.

The mean TLV and SII/III volume for all scans were 1705 ± 462 mL and 304 ± 115 mL for Synapse 3D, 1672 ± 463 mL and 282 ± 109 mL for Syngo.via and 1627 ± 449 mL and 277 ± 108 mL for OsiriX^®, respectively. Mean (percent) differences in TLV and SII/III are shown in Table 4. Mean TLV differences were significant for all software programs (P < 0.001). Regarding mean SII/III volumes, significant differences were observed between Syngo.via and Synapse 3D (P < 0.001), and between Synapse 3D and OsiriX^® (P = 0.001). Bland Altman analysis for OsiriX^®, Synapse 3D and Syngo.via showed good agreements with mean differences close to zero for both TLV and SII/III (Figures 2 and 3).

Figure 2 Bland Altman plots and scatter plot of the total liver volumes among the three software programs. A: Bland Altman plots of mean differences between Syngo.via and Synapse 3D; B: Scatter plot of mean differences between Syngo.via and Synapse 3D; C: Bland Altman plots of mean differences between Syngo.via and OsiriX^® for total liver volume; D: Scatter plot of mean differences between Syngo.via and OsiriX^®; E: Bland Altman plots of mean differences between Osirix^® and Synapse 3D; F: Scatter plot of mean differences between OsiriX^® and Synapse 3D. TLV: Total liver volume.

Figure 3 Bland Altman plots and graphs of segment II/III volumes among the three software programs. A: Bland Altman plots of mean differences between Syngo.via and Synapse 3D; B: Scatter plot of mean differences between Syngo.via and Synapse 3D; C: Bland Altman plots of mean differences between Syngo.via and OsiriX^® for total liver volume; D: Scatter plot of mean differences between Syngo.via and OsiriX^®; E: Bland Altman plots of mean differences between OsiriX^® and Synapse 3D; F: Scatter plot of mean differences between OsiriX^® and Synapse 3D. SII/III: Segment II/III.

Table 4 Differences in mean volumes among Syngo.via, OsiriX^® and Syngo.via for all scans, mean ± SD.

Item	Mean differences in mL	Mean percent differences	P value
TLV
OsiriX® vs Syngo.via	-45.3 ± 49.0	-2.6 ± 3.0	< 0.001
OsiriX® vs Synapse 3D	-78.5 ± 48.8	-3.9 ± 3.6	< 0.001
Syngo.via vs Synapse 3D	-33.2 ± 42.0	-1.7 ± 3.9	< 0.001
SII/III
OsiriX® vs Syngo.via	-4.4 ± 36.0	-0.8 ± 14.0	0.111
OsiriX® vs Synapse 3D	-26.2 ± 29.8	-8.5 ± 9.5	< 0.001
Syngo.via vs Synapse 3D	-21.8 ± 33.6	-6.8 ± 11.1	< 0.001

TLV: Total liver volume; SII/III: Segment II/III.

Mean FLR% for Synapse 3D, Syngo.via and OsiriX^® were 17.9% ± 4.9%, 16.9% ± 4.7% and 17.3% ± 4.8%, respectively, with significant differences between Synapse 3D and Syngo.via, and between Synapse 3D and OsiriX^® (P < 0.001). For sFLR, statistical differences were found between Synapse 3D and Syngo.via (18.6% ± 6.6% vs 17.2% ± 6.3%; P < 0.001), and between Synapse 3D and OsiriX^® (18.6% ± 6.6% vs 16.9% ± 5.9%; P < 0.001).

The mean time of segmentation for TLV and SII/III combined was 187 ± 72 seconds, 250 ± 99 seconds, and 518 ± 141 seconds for Synapse 3D, Syngo.via and OsiriX^®, respectively. Significant differences in segmentation time were found between OsiriX^® and Synapse 3D (P < 0.001), and between OsiriX^® and Syngo.via (P < 0.001).

DISCUSSION

In this study, two (semi-)automated software programs (Synapse 3D and Syngo.via) and one manual program (OsiriX^®) for liver volumetry were evaluated in healthy livers. By focusing on non-pathological liver parenchyma without tissue heterogeneity or presence of lesions, we aimed to assess the algorithmic performance of these software programs as thoroughly as possible. This approach allowed us to establish a baseline for comparison among the three software programs under ideal anatomical conditions, minimizing the influence of confounding factors. Our findings showed minor differences between manual delineation and (semi-)automated calculation of liver volume among the three used software programs. Manual assessment of liver volume calculations is often described as time-consuming and can result in incorrect FLR volumes for patients undergoing liver resection, possibly increasing the risk of posthepatectomy liver failure[18,19]. However, our findings suggest that all three software programs provide reliable liver volume calculations.

Excellent agreement between the observers was seen. However, significant differences in median volume of TLV and SII/III were observed, indicating a certain level of interobserver variation. Taking the experience of the two observers into account, a learning curve might also have influenced the results. Although a significant difference was found for FLR%, the absolute difference between the observers ranged from 0% to 2% across the software programs.

Looking at differences among the software programs, excellent agreement was seen for both TLV and SII/III volume with all three software programs. The widest 95%CIs were seen for Synapse 3D vs OsiriX^® for both TLV (95%CI: 0.655-0.997) and SII/III (95%CI: 0.715-0.975), indicating the largest individual variation. This may be attributable to differences in the segmentation approach (manual vs semi-automatic), the underlying segmentation algorithms, and the study design. As this was a single-center study in which identical imaging parameters and CT acquisition protocols were used, internal validity was likely enhanced. This possibly also explains the higher percent differences for SII/III compared to TLV between Synapse 3D and OsiriX^® or Syngo.via in our study. Mean TLV and SII/III volumes significantly differed between Synapse 3D and OsiriX^®/Syngo.via, with Synapse 3D consistently showing the highest mean estimations, and OsiriX^® the lowest. The consistently higher measured volume in Synapse 3D is unlikely to be clinically relevant when the same software is used for consecutive measurements within a patient. However, when different software programs are used interchangeably to calculate the sFLR, one should be aware of a systematically higher volume, as this can result in overestimation of the FLR volume. This may affect clinical decision-making, particularly when the FLR is borderline for safe surgery. However, when comparing the FLR% between the programs, a systematic difference is normalized, as it is present in both the FLR and TLV volumes. In our analyses, this reduced the difference to 1%-2% between Synapse 3D and OsiriX^® or Syngo.via. Although this result was statistically significant, it can be questioned whether this small percent difference of 1%-2% is clinically relevant. These findings imply that all three software programs are reliable to use for liver volumetry prior to hepatectomy. In terms of time efficiency, Synapse 3D is most beneficial, followed by Syngo.via. As Synapse 3D has a more automatic approach to liver vessel and contour delineation, extensive manual adjustments for TLV and SII/III are reduced, resulting in decreased volumetry time when compared to Syngo.via and OsiriX^®.

Similar results have been demonstrated in a study conducted by Maki et al[12], in which Syngo.via, Synapse 3D, and manual volumetry were compared for FLR in 30 patients who underwent right hepatectomy for colorectal liver metastases. Volumetry was performed by three observers. FLR volume was significantly smaller in manual tracing compared to Synapse 3D (30.1% vs 32.0%; P < 0.002) and Syngo.via (30.1% vs 32.0%; P < 0.001), and the longest volumetry time was observed in manual delineations. In addition to these similar results regarding the FLR volume and volumetry time, our study also showed smaller liver volumes when manual volumetry was performed. Smaller volumes observed following manual delineation can be attributed to physicians possibly drawing contours within the true liver boundaries to avoid including extra-hepatic tissue or low-attenuation tissue. On the contrary, (semi-)automated software programs may use attenuation thresholds in which other structures with similar Hounsfield Units are included because of less clearly defined boundaries.

In Maki et al[12], the FLR varied in composition, making it more representative of differences in clinical settings. In contrast, our study consistently used SII/III as a surrogate for the FLR, aiding in the identification of systematic differences between software programs. It is therefore important to note that discrepancies in FLR measurements among programs may increase in clinical settings when segmentation is performed for an FLR other than SII/III, and on anatomically challenging livers such as those where multiple lesions are present and surgical history complicates accurate liver volumetry. This is especially important among patients with borderline sFLR requiring preoperative regenerative procedures such as portal vein embolization prior to hepatic surgery.

This study has limitations. First, the assessment of consistent performance across the full dataset may be influenced by the fact that only the first 30 patient scans were evaluated by both observers for interobserver variability, while the remaining 170 scans were divided between the two researchers. In their paper, Koo and Li[15] recommended assessing reliability using at least 30 cases evaluated by a minimum of three raters. Although our study was conducted with only two observers, the sample size used for interobserver reliability was sufficient. Second, only abdominal CT images with specific reconstructions and 3 mm thick slices derived from CT scanners of one vendor were included. As image quality and reconstruction characteristics can vary across vendors and protocols, this can limit the generalizability of our findings. Nevertheless, this standardization enhances internal validity by minimizing variability in image acquisition, resulting in a more controlled and systematic comparison of the performances of the three software programs. Third, inclusion of patients without liver pathology cannot fully encompass real-world challenges during pre-operative FLR volume assessment, such as presence of liver pathology or previous local liver therapies. It is known that liver pathologies such as steatosis increase liver volume[20]. Lastly, no external ground truth (i.e. surgical specimen) was available, as this is not ethical in this research population. By using healthy livers, however, we avoided manual delineation errors as much as possible for TLV.

Future research should expand on our findings by evaluating liver segmentation software performance in patients with liver pathologies such as steatosis or cirrhosis, where tissue heterogeneity and irregular morphology could affect volumetric accuracy and the learning curve. Since this study focused on healthy livers, the excellent interobserver and intraobserver variability may not directly translate to diseased livers. Therefore, we are unable to determine the most accurate method for liver volumetry before liver surgery in this patient group. On the other hand, this study has some notable strengths. First, the use of a randomly selected large patient cohort minimizes selection bias and enhances the generalizability of the study. Second, CT imaging in the portal venous phase with 3 mm slices were consistently used with similar scanning protocols. Additionally, the segmentation analyses were conducted in a blind manner, with both observers unaware of the patients’ baseline characteristics. This approach reduces the risk of observer bias, thereby strengthening the validity and reliability of the results. Moreover, three different software programs were compared in a large sample size, improving generalizability of the results.

CONCLUSION

All three software programs can be utilized for liver volumetry prior to hepatectomy depending on the clinician’s preference. However, with regard to time efficiency, Synapse 3D outperformed the other two software programs. Absolute volume measurements varied among the software programs, highlighting the importance of using FLR% or sFLR in clinical decision-making, as these metrics remained consistent across all three software programs.