Comparison of dagnostic performance between senior and junior radiologists using subjective assessment vs ovarian-adnexal reporting and data system

Abstract

BACKGROUND

The diagnosis of ovarian-adnexal lesions is challenging due to the variability in radiologist experience, which may lead to inconsistent diagnostic outcomes. The ovarian-adnexal reporting and data system (O-RADS) aims to standardize ultrasound-based diagnosis; however, optimization of the clinical utility of O-RADS remains unclear.

AIM

To compare the diagnostic performance of subjective assessment and O-RADS between senior and junior radiologists in ovarian adnexal lesions.

METHODS

This was a retrospective study and included 364 patients with 414 ovarian-adnexal lesions (30 malignant and 384 benign), all confirmed by pathological examination. A senior radiologist and a junior radiologist independently diagnosed all lesions using subjective assessment and O-RADS, respectively. Diagnostic indices including sensitivity, specificity, and accuracy were compared, and the consistency of O-RADS classification between the two radiologists was analyzed.

RESULTS

For the senior radiologist, the accuracy of subjective assessment (96.6%, 400/414) was significantly higher than that of O-RADS (89.1%, 369/414); compared with subjective assessment, O-RADS significantly decreased specificity (97.9% vs 90.1%, χ² = 20.812, P < 0.001) and accuracy (χ² = 17.538, P < 0.001), while sensitivity showed no significant change (80.0% vs 76.7%, χ² = 0.098, P = 0.754). For the junior radiologist, O-RADS significantly improved sensitivity compared with subjective assessment (76.7% vs 46.7%, χ² = 5.711, P = 0.017), with no significant decreases in specificity (95.3% vs 97.1%, χ² = 1.756, P = 0.185) or accuracy (94.0% vs 93.4%, χ² = 0.082, P = 0.775). The O-RADS classification consistency between the senior and junior radiologist was moderate (Kappa = 0.475, P < 0.001).

CONCLUSION

Senior radiologists should prioritize subjective assessment to maximize diagnostic accuracy, while junior radiologists can significantly benefit from O-RADS to improve sensitivity in diagnosing malignant lesions.

Key Words: Ovarian-adnexal reporting and data system; Ultrasound; Ovarian tumor; Diagnostic accuracy; Radiologist

Core Tip: For ovarian-adnexal lesion diagnosis, senior radiologists achieve higher accuracy with subjective assessment, while junior radiologists can significantly improve sensitivity in diagnosing malignant lesions using the ovarian-adnexal reporting and data system (O-RADS). Moderate O-RADS inter-radiologist consistency highlights the need for strengthened training to enhance diagnostic standardization, guiding personalized application of O-RADS based on radiologist experience.

INTRODUCTION

Ovarian cancer is one of the most common gynecological malignancies. In 2022, there were 324600 new cases and 207000 deaths globally with 61100 new cases and 32600 deaths reported in China. The mortality rate of ovarian cancer is the highest among gynecological malignancies[1,2]. Ovarian tumors have diverse classifications and complex histopathological sub-types[3].

Preoperative diagnosis of ovarian tumors mainly relies on imaging techniques for localization, qualitative diagnosis, and differential diagnosis. Among these modalities, ultrasound examination offers advantages including safety, non invasiveness, ease of operation, low cost, and repeatedly use. It serves as the preferred imaging method for ovarian tumors and plays an essential role in their diagnosis[4].

The homogenization and standardization of ultrasound examinations for ovarian tumors are of paramount importance. The international ovarian tumor analysis (IOTA) group standardized the terminology, definitions and measurements for ovarian tumors in 2000[5], and has subsequently developed several diagnostic models, including the LR1 model, the LR2 model, Simple Rules, Easy Descriptors and the Assessment of Different Neoplasia in the Adnexa model[6-9]. The American College of Radiology (ACR) established the ovarian-adnexal reporting and data system (O-RADS) committee in 2015; this committee issued a white paper on ovarian-adnexal reporting lexicon for ultrasound in 2018[10] and a consensus guideline on risk stratification and management of ovarian adnexal lesions in 2020[11].

Despite the extensive clinical application and considerable value of the IOTA diagnostic models and O-RADS stratification and management in the diagnosis and differential diagnosis of ovarian adnexal lesions, subjective assessment by ultrasound experts remains the most accurate[12,13]. Subjective assessment of adnexal masses using ultrasonography was first introduced by Timmerman et al[14]. These investigators did not provide any predefined criteria for malignancy based on ultrasound features, such as lesion morphology and vascularity. Instead, each investigator analyzed the images and classified masses as benign or malignant based on individual experience.

Thus, how to optimize the clinical utility of O-RADS remains unclear. In the present study, we retrospectively analyzed ultrasound images of ovarian-adnexal lesions independently by a junior radiologist and a senior radiologist separately. Each radiologist diagnosed all ovarian-dnexal lesions using subjective assessment and O-RADS, respectively. The aim of this study was to compare the diagnostic accuracy between the two radiologists and identify optimal approaches for the differential diagnosis of ovarian-adnexal lesions.

MATERIALS AND METHODS

Setting and ethics

This was a retrospective study approved by the ethics committee of our hospital (CNFBLLAR-2024-006). Written informed consent was obtained from all patients prior to ultrasound examinations.

Patients

Patients who underwent adnexal surgery for ovarian adnexal lesions at our hospital from January 2024 to February 2025 were included in this study. The inclusion criteria were as follows: (1) Patients with a confirmed postoperative pathological diagnosis of ovarian-adnexal lesions; and (2) Patients who underwent gynecological ultrasound examination within 1 week before adnexal surgery. The exclusion criteria were as follows: (1) No ovarian-adnexal lesions detected by preoperative ultrasound examinations; (2) Patients who refused to provide informed consent for the use of their clinical and imaging data in this study; and (3) Ovarian-adnexal lesions not evaluated, measured or recorded in accordance with the white paper and consensus guideline of O-RADS committee.

Ultrasound examinations

All the ultrasound examinations were performed by radiologists with at least 5 years of experience in gynecological ultrasound practice. Prior to the study, these radiologists received training on IOTA consensus[5] and O-RADS consensus guideline[11]. During each examination, ovarian adnexal lesions were systematically observed, measured, and described accordingly and all the ultrasound images were saved for subsequent review. The ultrasound devices used in this study included three types of diagnostic systems, each equipped with corresponding transvaginal and transabdominal probes: (1) Resona R9 (Mindray Medical International, Shenzhen, Guangdong Province, China): Transvaginal probe V11-3HU and transabdominal probe SC6-1U; (2) UGEO WS80A (Samsung, Seoul, Korea): Transvaginal probe VR 5-9 and transabdominal probe SC1-6; and (3) Voluson E8 (GE HealthCare, CA, United States): Transvaginal probe IC5-9-D and transabdominal probe C1-5.

Retrospective analysis of ovarian adnexal lesions

A junior radiologist (Zhu Q) with 7 years’ experience of gynecological ultrasound examination and a senior radiologist (Wu JJ) with 20 years’ experience of gynecological ultrasound examination reviewed all the patients retrospectively, including patients’ age, ultrasound images of ovarian adnexal lesions and ultrasound description of ovarian adnexal lesions. The two radiologists were blinded to the patients’ pathological results and other test results except ultrasound examinations. They were asked to assess the lesions subjectively as benign or malignant first; and then classify the lesions according to O-RADS guideline[11].

Statistical analysis

SPSS version 24.0 software (IBM Corporation, Chicago, IL, United States) was used for statistical analysis. P < 0.05 was considered statistically significant. Numerical variables are presented as mean ± SD. Kolmogorov-Smirnov test was used to determine whether they conformed to a normal distribution. If not, nonparametric Mann-Whitney test was used to compare between the groups. Enumeration data are presented as n (%) and compared among the groups using the χ² test. Cohen’s Kappa test was used for the analysis of the intra-observer consistency of O-RADS classification.

RESULTS

General information of the patients

A total of 364 patients and 414 ovarian-adnexal lesions were included in this study. Among these lesions, 30 were confirmed as malignant and 384 as benign by pathological diagnosis. Malignant lesions consisted of 14 serous borderline tumors, one mucinous borderline tumor, two endometrioid carcinomas, eight high-grade serous carcinomas, one low-grade serous carcinoma, one ovarian carcinoid combined with cystic teratoma, one squamous cell carcinoma, two serous fallopian carcinomas. The benign lesions included 113 benign cystic teratomas, 193 endometriomas, 35 serous cystadenomas, 29 mucinous cystadenomas, two seromucinous cystadenomas, two corpus luteum cysts, one Brenner tumor, six simple cysts, one adenofibroma and two fibromas.

Among the 364 patients, 314 had unilateral lesions and 50 had bilateral lesions. For one patient with both a benign and a malignant bilateral lesion, the patient was classified into the malignant group. A significant difference was observed in age between the benign group (n = 340) and the malignant group (n = 24) (39.53 ± 11.25 vs 50.67 ± 14.60, Z = 3.628, P < 0.001).

Comparison of diagnostic accuracy between the senior and junior radiologists using subjective assessment

The sensitivity, specificity, and accuracy of the senior radiologist using subjective assessment were 80.0% (24/30), 97.9% (376/384), and 96.6% (400/414), respectively. The sensitivity, specificity, and accuracy of the junior radiologist using subjective assessment were 46.7% (14/30), 97.1% (373/384), and 93.4% (387/414), respectively.

Compared with the junior radiologist using subjective assessment, the senior radiologist achieved significantly higher sensitivity (χ² = 7.177, P = 0.007) and accuracy (χ² = 4.337, P = 0.037), while the specificity was comparable between the two radiologists (χ² = 0.486, P = 0.486).

Comparison of diagnostic accuracy between subjective assessment and O-RADS for the senior radiologist

The sensitivity, specificity, and accuracy of the senior radiologist using O-RADS were 76.7% (23/30), 90.1% (346/384) and 89.1% (369/414). When the senior radiologist used O-RADS for diagnosing adnexal lesions, both the specificity (χ² = 20.812, P = 0.000) and accuracy (χ² = 17.538, P < 0.001) decreased significantly, while the sensitivity showed a non-significant decrease (c² = 0.098, P = 0.754).

Comparison of diagnostic accuracy between subjective assessment and O-RADS for the junior radiologist

The sensitivity, specificity, and accuracy of the junior radiologist using O-RADS were 76.7% (23/30), 95.3% (366/384) and 94.0% (389/414). When the junior radiologist used O-RADS for diagnosing adnexal lesions, the sensitivity was significantly improved (χ² = 5.711, P = 0.017), while the specificity (χ² = 1.756, P = 0.185) and accuracy (χ² = 0.082, P = 0.775) did not decrease significantly.

Diagnostic accuracy for borderline tumors

For the senior radiologist, the sensitivity using O-RADS (80%, 12/15) for borderline tumors and the sensitivity using subjective assessment (86.7%, 13/15) showed no significant difference (χ² = 0.240, P = 1.00). For the junior radiologist, the sensitivity using O-RADS (93.3%, 14/15) for borderline tumors was significantly improved, compared with using subjective assessment (20%, 3/15) (χ² = 16.425, P < 0.001).

Consistency of O-RADS classification between the senior and junior radiologists

The O-RADS classifications of the senior and junior radiologists are presented in Table 1 and Figure 1. The O-RADS classifications by the two radiologists showed moderate consistency (Kappa = 0.475, P < 0.001).

Figure 1 Pathologically confirmed serous borderline cystadenoma with inconsistent ovarian-adnexal reporting and data system classifications between the senior and junior radiologist. A: Unilocular-solid cyst with a smooth internal wall, low-level cystic content, and a solid papillary projection; B: Minimal blood flow detected in the papillary projection (score = 2). The junior radiologist classified the lesion as ovarian-adnexal reporting and data system (O-RADS) 2 and diagnosed it as a benign cystic teratoma, whereas the senior radiologist classified it as O-RADS 5 and diagnosed it as serous cystadenocarcinoma.

Table 1 Ovarian-adnexal reporting and data system classification consistency between the senior and junior radiologist (moderate consistency).

O-RADS classification by the senior radiologist. O-RADS classification by the junior radiologist	2	3	4	5	Total
0	7	0	0	1	8
2	297	6	0	1	304
3	22	15	1	2	40
4	20	4	14	10	48
5	3	0	3	8	14
Total	349	25	18	22	414

O-RADS: Ovarian-adnexal reporting and data system.

DISCUSSION

This study systematically compared the diagnostic performance of subjective assessment and O-RADS between a senior and junior radiologist for ovarian-adnexal lesions, and further analyzed the consistency of O-RADS classification between the two radiologists. The key findings revealed that subjective assessment by the senior radiologist yielded higher diagnostic accuracy than O-RADS, while the junior radiologist achieved significantly improved sensitivity without compromising specificity or accuracy after applying O-RADS. These results provide valuable insights for optimizing the clinical application of ultrasound diagnosis in ovarian-adnexal lesions, especially for radiologists with different levels of experience.

The superior diagnostic accuracy of the senior radiologist’s subjective assessment over O-RADS is consistent with the findings of previous studies[12,13], which emphasized that the accumulated clinical experience of senior radiologists enables comprehensive judgment of complex imaging features that may not be fully captured by standardized diagnostic systems. In this study, the senior radiologist achieved a high diagnostic accuracy of 96.6% through subjective assessment, which was significantly higher than the 89.1% accuracy when using O-RADS. This suggests that standardized systems such as O-RADS may not fully reflect the diagnostic value of senior radiologists’ experience, and excessive adherence to rigid criteria may even reduce diagnostic performance.

In contrast, the application of O-RADS significantly improved the diagnostic sensitivity of the junior radiologist (46.7% vs 76.7%, χ² = 5.711, P = 0.017), especially for borderline tumors (20% vs 93.3%, χ² = 16.425, P < 0.001), without significant decreases in specificity (97.1% vs 95.3%, χ² = 1.756, P = 0.185) or accuracy (93.4% vs 94.0%, χ² = 0.082, P = 0.775). This finding aligns well with the original purpose of O-RADS establishment-not only to standardize ultrasound reporting and improve diagnostic consistency among radiologists[10,11], but also to enhance the sensitivity of ovarian cancer diagnosis, reduce missed diagnoses of malignant lesions, and thereby provide patients with ovarian cancer the opportunity for early treatment[15,16]. Junior radiologists often lack experience in identifying malignant imaging features, leading to underdiagnosis of malignant lesions (low sensitivity) in subjective assessment. O-RADS provides a structured evaluation framework that guides junior radiologists to systematically assess key diagnostic indicators, thereby reducing missed diagnoses of malignant lesions[17-19].

The clinical implications of this study are noteworthy. For senior radiologists, subjective assessment should be the primary diagnostic method, while O-RADS can be used as an auxiliary tool to standardize reporting and avoid missed diagnoses caused by subjective bias. For junior radiologists, O-RADS is recommended as a routine diagnostic guide to improve the sensitivity of malignant lesion detection and ensure diagnostic standardization. Furthermore, the moderate consistency of O-RADS classification suggests that targeted training on O-RADS criteria may further improve the consistency between radiologists of different experience levels, which is of great significance for promoting the homogenization of ovarian ultrasound diagnosis.

This study has several limitations that should be acknowledged. First, as a retrospective study, the diagnostic analysis relied on ultrasound images previously stored before and not real-time ultrasound examinations. This may result in incomplete or even wrong information acquisition and lead to result bias. Second, this was a single-center study with a relatively small sample size of malignant lesions (n = 30), which may limit the generalizability of the results. Future prospective multicenter studies with larger sample sizes are needed to validate the findings.

CONCLUSION

The diagnostic value of subjective assessment and O-RADS varies with radiologist experience. Senior radiologists should prioritize subjective assessment to maximize diagnostic accuracy, while junior radiologists can significantly benefit from O-RADS to improve sensitivity in diagnosing malignant lesions. The moderate consistency of O-RADS classification suggests the need to strengthen its clinical promotion and training to enhance standardization of ovarian-adnexal lesion diagnosis.