Interobserver agreement for contrast-enhanced ultrasound of liver imaging reporting and data system: A systematic review and meta-analysis

Table 1 Characteristics of included studies

Ref.	Publication year	Country	Study design	Enrollment method	US system	Number of patients (male/female)	Age in yr, mean ± SD	Number of FLLs (benign/malignant)	LI-RADS version	Reference standard
Chen et al[25]	2017	China	Retrospective	Consecutive	Hitachi HI Vision Preirus; Siemens S3000	377 (207/170)	43.56 11.48	429 (96/333)	2016	Pathology and SCRS
Schellhaas et al[20]	2018	Germany	Retrospective	Consecutive	Siemens S2000; GE Logiq E9; Toshiba Aplio 500	55 (44/11)	65.90 (53-86)1	55 (5/50)	2016	Pathology and SCRS
Ling et al[28]	2018	China	Retrospective	Consecutive	Philips IU22	56 (44/12)	52.52	56 (10/46)	2016	Pathology
Liu et al[26]	2019	China	Prospective	Consecutive	Philips IU22; Toshiba Aplio 500	82 (61/21)	58.15 10.97	82 (34/48)	2017	Pathology and SCRS
Zhou et al[27]	2019	China	Retrospective	Consecutive	GE Logiq E9	241 (166/75)	56 10	272 (44/228)	2016	Pathology and SCRS
Tan et al[21]	2020	Singapore	Retrospective	Selective	GE Logiq E9; Toshiba Aplio 500	45 (32/13)	63.1 (34-84)	46 (9/37)	2017	Pathology and SCRS
Wang et al[6]	2020	China	Retrospective	Consecutive	Hitachi Hi Vision Avius	258 (200/58)	52 11	355 (235/120)	2017	Pathology and SCRS
Cui et al3	2020	China	Retrospective	Selective	Hitachi Hi Vision Avius	63 (47/16)	56 8	84 (35/49)	2017	Pathology and SCRS

¹This study only included age range and mean.

²This study used the median of age.

³This is our study. SCRS: Synthesized clinical reference standard; FLLS: Focal liver lesions; LI-RADS: Liver Imaging Reporting and Data System; SD: Standard deviation; US: Ultrasound.

Table 2 Analysis of interobserver agreement for included studies

Ref.	Kappa value (95%CI)	Weight in fixed model, %	Weight in random model, %
Chen et al[25]	0.81 (0.77, 0.84)	31.40	14.00
Schellhaas et al[20]	0.39 (0.14, 0.59)	3.80	11.40
Ling et al[28]	0.69 (0.52, 0.81)	3.90	11.40
Liu et al[26]	0.61 (0.45, 0.73)	5.80	12.30
Zhou et al[27]	0.87 (0.84, 0.90)	19.90	13.80
Tan et al[21]	0.94 (0.89, 0.97)	3.20	10.90
Wang et al[6]	0.77 (0.72, 0.81)	26.00	13.90
Cui et al1	0.67 (0.53, 0.77)	6.00	12.30

¹This is our study. CI: Confidence interval.

Table 3 Interobserver agreement of pooled included studies

Summary type	Kappa value (95%CI)	Z	P value
Fixed effects model	0.79 (0.77, 0.81)	39.68	< 0.01
Random effects model	0.76 (0.67, 0.83)	9.99	< 0.01

CI: Confidence interval.

Table 4 Sensitivity analysis eliminating studies one by one

Ref.	Kappa value (95%CI)	I2 (95%CI), %	P value
Chen et al[25]	0.75 (0.62; 0.84)	92.42 (87.07; 95.57)	< 0.01
Schellhaas et al[20]	0.79 (0.71; 0.85)	89.44 (80.83; 94.22)	< 0.01
Ling et al[28]	0.77 (0.67; 0.84)	92.30 (86.72; 95.58)	< 0.01
Liu et al[26]	0.78 (0.69; 0.85)	91.44 (84.80; 95.13)	< 0.01
Zhou et al[27]	0.74 (0.63; 0.82)	89.78 (81.61; 94.36)	< 0.01
Tan et al[21]	0.73 (0.62; 0.80)	90.21 (82.51; 94.60)	< 0.01
Wang et al[6]	0.76 (0.64; 0.85)	92.41 (86.90; 95.59]	< 0.01
Cui et al1	0.77 (0.67; 0.85)	92.04 (86.02; 95.43)	< 0.01

¹This is our study. CI: Confidence interval.

Table 5 Results of meta-regression analysis of interobserver agreement of Liver Imaging Reporting and Data System in contrast-enhanced ultrasound

Covariate and subgroup	Number of included studies	Kappa value (95%CI)	P value
Study design			0.26
Retrospective	7	0.78 (0.69, 0.85)
Prospective	1	0.61 (0.45, 0.73)
Enrollment method			< 0.01
Consecutive	7	0.73 (0.62, 0.80)
Selective	1	0.94 (0.89, 0.97)
LI-RADS version			0.66
2016	4	0.74 (0.58, 0.85)
2017	4	0.78 (0.62, 0.88)
Number of interobservers			< 0.01
2	7	0.79 (0.71, 0.85)
3	1	0.39 (0.14, 0.59)
Number of US systems			0.93
1	4	0.77 (0.65, 0.85)
> 1	4	0.76 (0.51, 0.89)
Race of patients			< 0.01
Westerners	1	0.79 (0.71, 0.85)
Easterners	7	0.39 (0.14, 0.59)
Number of FLLs			0.16
< 100	5	0.72 (0.48, 0.86)
> 100	3	0.82 (0.76, 0.87)
Reference standard			0.58
Pathology	1	0.69 (0.52, 0.81)
Pathology and SCRS	7	0.77 (0.67, 0.84)

FLL: Focal hepatic lesion; LI-RADS: Liver Imaging Reporting and Data System; SCRS: Synthesized clinical reference standard; US: Ultrasound.