Retrospective Cohort Study
Copyright ©The Author(s) 2022. Published by Baishideng Publishing Group Inc. All rights reserved.
World J Radiol. Jan 28, 2022; 14(1): 19-29
Published online Jan 28, 2022. doi: 10.4329/wjr.v14.i1.19
Effect of training on resident inter-reader agreement with American College of Radiology Thyroid Imaging Reporting and Data System
Yang Du, Meredith Bara, Prayash Katlariwala, Roger Croutze, Katrin Resch, Jonathan Porter, Medica Sam, Mitchell P Wilson, Gavin Low
Yang Du, Meredith Bara, Prayash Katlariwala, Roger Croutze, Katrin Resch, Jonathan Porter, Medica Sam, Mitchell P Wilson, Gavin Low, Department of Radiology and Diagnostic Imaging, University of Alberta, Edmonton T6G 2B7, Alberta, Canada
Author contributions: Du Y, Bara M and Low G designed the study; Du Y, Bara M, Croutze R, Resch K, Porter J, Sam M, Wilson MP and Low G performed the research; Du Y, Bara M, Katlariwala P, Low G and Wilson MP analyzed the data and wrote the manuscript; all authors have read and approved the final manuscript.
Institutional review board statement: This retrospective, single-institution observational study was approved by the institutional Health Research Ethics Board (Pro 00104708).
Informed consent statement: This study was exempted from obtaining informed consent.
Conflict-of-interest statement: The authors have no conflict of interest to declare.
Data sharing statement: The raw dataset is available from the corresponding author at yang.du@usask.ca. Consent for data sharing was not obtained but the presented data are anonymized and risk of identification is low.
STROBE statement: Guidelines of the STROBE statement have been adopted.
Open-Access: This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: https://creativecommons.org/Licenses/by-nc/4.0/
Corresponding author: Yang Du, BSc, FRCPC, MD, Doctor, Staff Physician, Department of Radiology and Diagnostic Imaging, University of Alberta, 2A2.41 WMC, 8440-112 St NW, Edmonton T6G 2B7, Alberta, Canada. yang.du@usask.ca
Received: October 12, 2021
Peer-review started: October 12, 2021
First decision: December 9, 2021
Revised: December 21, 2021
Accepted: January 11, 2022
Article in press: January 11, 2022
Published online: January 28, 2022
Processing time: 101 Days and 12.4 Hours
Abstract
BACKGROUND

The American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) was introduced to standardize the ultrasound characterization of thyroid nodules. Studies have shown that ACR-TIRADS reduces unnecessary biopsies and improves consistency of imaging recommendations. Despite its widespread adoption, there are few studies to date assessing the inter-reader agreement amongst radiology trainees with limited ultrasound experience. We hypothesize that in PGY-4 radiology residents with no prior exposure to ACR TI-RADS, a statistically significant improvement in inter-reader reliability can be achieved with a one hour training session.

AIM

To evaluate the inter-reader agreement of radiology residents in using ACR TI-RADS before and after training.

METHODS

A single center retrospective cohort study evaluating 50 thyroid nodules in 40 patients of varying TI-RADS levels was performed. Reference standard TI-RADS scores were established through a consensus panel of three fellowship-trained staff radiologists with between 1 and 14 years of clinical experience each. Three PGY-4 radiology residents (trainees) were selected as blinded readers for this study. Each trainee had between 4 to 5 mo of designated ultrasound training. No trainee had received specialized TI-RADS training prior to this study. Each of the readers independently reviewed the 50 testing cases and assigned a TI-RADS score to each case before and after TI-RADS training performed 6 wk apart. Fleiss kappa was used to measure the pooled inter-reader agreement. The relative diagnostic performance of readers, pre- and post-training, when compared against the reference standard.

RESULTS

There were 33 females and 7 males with a mean age of 56.6 ± 13.6 years. The mean nodule size was 19 ± 14 mm (range from 5 to 63 mm). A statistically significant superior inter-reader agreement was found on the post-training assessment compared to the pre-training assessment for the following variables: 1. “Shape” (k of 0.09 [slight] pre-training vs 0.67 [substantial] post-training, P < 0.001), 2. “Echogenic foci” (k of 0.28 [fair] pre-training vs 0.45 [moderate] post-training, P = 0.004), 3. ‘TI-RADS level’ (k of 0.14 [slight] pre-training vs 0.36 [fair] post-training, P < 0.001) and 4. ‘Recommendations’ (k of 0.36 [fair] pre-training vs 0.50 [moderate] post-training, P = 0.02). No significant differences between the pre- and post-training assessments were found for the variables 'composition', 'echogenicity' and 'margins'. There was a general trend towards improved pooled sensitivity with TI-RADS levels 1 to 4 for the post-training assessment while the pooled specificity was relatively high (76.6%-96.8%) for all TI-RADS level.

CONCLUSION

Statistically significant improvement in inter-reader agreement in the assigning TI-RADS level and recommendations after training is observed. Our study supports the use of dedicated ACR TI-RADS training in radiology residents.

Keywords: Thyroid; Thyroid nodule; American College of Radiology Thyroid Imaging Reporting and Data System; Inter-reader agreement; Ultrasound

Core Tip: There is a statistically significant improvement in inter-reader agreement among radiology trainees with limited ultrasound experience using the American College of Radiology Thyroid Imaging Reporting and Data System (TI-RADS) after training for TI-RADS grading and recommendations. This study demonstrates the learnability of TI-RADS in radiology trainees.