1
|
Leutz-Schmidt P, Palm V, Mathy RM, Grözinger M, Kauczor HU, Jang H, Sedaghat S. Performance of Large Language Models ChatGPT and Gemini on Workplace Management Questions in Radiology. Diagnostics (Basel) 2025; 15:497. [PMID: 40002648 PMCID: PMC11854386 DOI: 10.3390/diagnostics15040497] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2024] [Revised: 02/17/2025] [Accepted: 02/17/2025] [Indexed: 02/27/2025] Open
Abstract
Background/Objectives: Despite the growing popularity of large language models (LLMs), there remains a notable lack of research examining their role in workplace management. This study aimed to address this gap by evaluating the performance of ChatGPT-3.5, ChatGPT-4.0, Gemini, and Gemini Advanced as famous LLMs in responding to workplace management questions specific to radiology. Methods: ChatGPT-3.5 and ChatGPT-4.0 (both OpenAI, San Francisco, CA, USA) and Gemini and Gemini Advanced (both Google Deep Mind, Mountain View, CA, USA) generated answers to 31 pre-selected questions on four different areas of workplace management in radiology: (1) patient management, (2) imaging and radiation management, (3) learning and personal development, and (4) administrative and department management. Two readers independently evaluated the answers provided by the LLM chatbots. Three 4-point scores were used to assess the quality of the responses: (1) overall quality score (OQS), (2) understandabilityscore (US), and (3) implementability score (IS). The mean quality score (MQS) was calculated from these three scores. Results: The overall inter-rater reliability (IRR) was good for Gemini Advanced (IRR 79%), Gemini (IRR 78%), and ChatGPT-3.5 (IRR 65%), and moderate for ChatGPT-4.0 (IRR 54%). The overall MQS averaged 3.36 (SD: 0.64) for ChatGPT-3.5, 3.75 (SD: 0.43) for ChatGPT-4.0, 3.29 (SD: 0.64) for Gemini, and 3.51 (SD: 0.53) for Gemini Advanced. The highest OQS, US, IS, and MQS were achieved by ChatGPT-4.0 in all categories, followed by Gemini Advanced. ChatGPT-4.0 was the most consistently superior performer and outperformed all other chatbots (p < 0.001-0.002). Gemini Advanced performed significantly better than Gemini (p = 0.003) and showed a non-significant trend toward outperforming ChatGPT-3.5 (p = 0.056). ChatGPT-4.0 provided superior answers in most cases compared with the other LLM chatbots. None of the answers provided by the chatbots were rated "insufficient". Conclusions: All four LLM chatbots performed well on workplace management questions in radiology. ChatGPT-4.0 outperformed ChatGPT-3.5, Gemini, and Gemini Advanced. Our study revealed that LLMs have the potential to improve workplace management in radiology by assisting with various tasks, making these processes more efficient without requiring specialized management skills.
Collapse
Affiliation(s)
- Patricia Leutz-Schmidt
- Department of Diagnostic and Interventional Radiology, University Hospital Heidelberg, 69120 Heidelberg, Germany; (P.L.-S.); (V.P.); (R.M.M.); (H.-U.K.)
| | - Viktoria Palm
- Department of Diagnostic and Interventional Radiology, University Hospital Heidelberg, 69120 Heidelberg, Germany; (P.L.-S.); (V.P.); (R.M.M.); (H.-U.K.)
| | - René Michael Mathy
- Department of Diagnostic and Interventional Radiology, University Hospital Heidelberg, 69120 Heidelberg, Germany; (P.L.-S.); (V.P.); (R.M.M.); (H.-U.K.)
| | - Martin Grözinger
- German Cancer Research Center (DKFZ), 69120 Heidelberg, Germany;
| | - Hans-Ulrich Kauczor
- Department of Diagnostic and Interventional Radiology, University Hospital Heidelberg, 69120 Heidelberg, Germany; (P.L.-S.); (V.P.); (R.M.M.); (H.-U.K.)
| | - Hyungseok Jang
- Department of Radiology, University of California Davis, Davis, CA 95616, USA;
| | - Sam Sedaghat
- Department of Diagnostic and Interventional Radiology, University Hospital Heidelberg, 69120 Heidelberg, Germany; (P.L.-S.); (V.P.); (R.M.M.); (H.-U.K.)
| |
Collapse
|
2
|
White DG, O'Brien MA, Cornacchi SD, Freeman R, Grunfeld E. COVID-19: A Qualitative Analysis of Academic Family Physician Leaders' Crisis Response. Fam Med 2023; 55:38-44. [PMID: 36656886 PMCID: PMC10681343 DOI: 10.22454/fammed.55.421082] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/07/2023]
Abstract
BACKGROUND AND OBJECTIVES The onset of the COVID-19 pandemic severely threatened all aspects of academic family medicine, constituting a crisis. Multiple publications have identified recommendations and documented the creative responses of primary care and academic organizations to address these challenges, but there is little research on how decisions came about. Our objective was to gain insight into the context, process, and nature of family medicine leaders' discussions in pivoting to address a crisis. METHODS We used a qualitative descriptive design to explore new dimensions of existing concepts. The setting was the academic family medicine department at the University of Toronto. To identify leadership themes, we used the constant comparative method to analyze transcripts of monthly meetings of the departmental executive: three meetings immediately before and three following the declaration of a state emergency in Ontario. RESULTS Six themes were evident before and after the onset of the pandemic: building capacity in academic family medicine; developing leadership; advancing equity, diversity, and inclusion; learner safety and wellness; striving for excellence; and promoting a supportive and collegial environment. Five themes emerged as specific responses to the crisis: situational awareness; increased multidirectional communication; emotional awareness; innovation in education and patient care; and proactive planning for extended adaptation to the pandemic. CONCLUSION Existing cultural and organizational approaches formed the foundation for the crisis response, while crisis-specific themes reflected skills and attitudes that are essential in clinical family medicine, including adapting to community needs, communication, and emotional awareness.
Collapse
Affiliation(s)
- David G. White
- Department of Family and Community Medicine, University of TorontoToronto, ONCanada
- North York General HospitalNorth York, ONCanada
| | - Mary Ann O'Brien
- Department of Family and Community Medicine, University of TorontoToronto, ONCanada
| | - Sylvie D. Cornacchi
- Department of Family and Community Medicine, University of TorontoToronto, ONCanada
| | - Risa Freeman
- Department of Family and Community Medicine, University of TorontoToronto, ONCanada
- North York General HospitalNorth York, ONCanada
| | - Eva Grunfeld
- Department of Family and Community Medicine, University of TorontoToronto, ONCanada
- Ontario Institute for Cancer ResearchToronto, ONCanada
| |
Collapse
|
3
|
Tay YX, Kothan S, Kada S, Cai S, Lai CWK. Challenges and optimization strategies in medical imaging service delivery during COVID-19. World J Radiol 2021; 13:102-121. [PMID: 34141091 PMCID: PMC8188837 DOI: 10.4329/wjr.v13.i5.102] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/10/2021] [Revised: 04/10/2021] [Accepted: 05/10/2021] [Indexed: 02/06/2023] Open
Abstract
In coronavirus disease 2019 (COVID-19), medical imaging plays an essential role in the diagnosis, management and disease progression surveillance. Chest radiography and computed tomography are commonly used imaging techniques globally during this pandemic. As the pandemic continues to unfold, many healthcare systems worldwide struggle to balance the heavy strain due to overwhelming demand for healthcare resources. Changes are required across the entire healthcare system and medical imaging departments are no exception. The COVID-19 pandemic had a devastating impact on medical imaging practices. It is now time to pay further attention to the profound challenges of COVID-19 on medical imaging services and develop effective strategies to get ahead of the crisis. Additionally, preparation for operations and survival in the post-pandemic future are necessary considerations. This review aims to comprehensively examine the challenges and optimization of delivering medical imaging services in relation to the current COVID-19 global pandemic, including the role of medical imaging during these challenging times and potential future directions post-COVID-19.
Collapse
Affiliation(s)
- Yi Xiang Tay
- Radiography Department, Singapore General Hospital, Singapore 169608, Singapore
| | - Suchart Kothan
- Center of Radiation Research and Medical Imaging, Department of Radiologic Technology, Faculty of Associated Medical Sciences, Chiang Mai University, Chiang Mai 50000, Thailand
| | - Sundaran Kada
- Faculty of Health and Social Sciences, Western Norway University of Applied Sciences, Bergen Postbox 7030, 5020 Bergen, Norway
| | - Sihui Cai
- Radiography Department, Singapore General Hospital, Singapore 169608, Singapore
| | - Christopher Wai Keung Lai
- Department of Health and Social Sciences, Singapore Institute of Technology, Singapore 138683, Singapore
| |
Collapse
|
4
|
Mossa-Basha M, Schnall M, Meltzer CC, Oleaga L, Filippi CG, Mahoney M, Donoso-Bach L, Tan BS. Special Report of the RSNA COVID-19 Task Force: Crisis Leadership of Major Health System Radiology Departments during COVID-19. Radiology 2021; 299:E187-E192. [PMID: 33399509 PMCID: PMC7993242 DOI: 10.1148/radiol.2020203518] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2020] [Revised: 10/28/2020] [Accepted: 11/25/2020] [Indexed: 12/28/2022]
Abstract
Severe acute respiratory syndrome coronavirus 2 has spread across the world since December 2019, infecting 100 million and killing millions. The impact on health care institutions during the coronavirus disease 2019 pandemic has been considerable, with exhaustion of institutional and personal protective equipment resources during local outbreaks and crushing financial consequences for many institutions. Establishing adaptive principles of leadership is necessary during crises, fostering quick decision-making and workflow modifications, while a rapid review of data must determine necessary course corrections. This report describes concepts of crisis leadership teams that can help maximize their effectiveness during the current and future pandemics.
Collapse
Affiliation(s)
- Mahmud Mossa-Basha
- From the Department of Radiology, University of Washington, 1959 NE Pacific St, Seattle, WA 98195 (M.M.B.); Department of Radiology, University of Pennsylvania, Philadelphia, Pa (M.S.); Department of Radiology and Imaging Sciences, Emory University, Atlanta, Ga (C.C.M.); Department of Radiology, Hospital Clínic, Barcelona, Spain (L.O., L.D.); Department of Radiology, Northwell Health, Manhasset, NY (C.G.F.); Department of Radiology, University of Cincinnati, Cincinnati, Ohio (M.M.); and Department of Vascular and Interventional Radiology, Singapore General Hospital, Singapore (B.S.T.)
| | - Mitchell Schnall
- From the Department of Radiology, University of Washington, 1959 NE Pacific St, Seattle, WA 98195 (M.M.B.); Department of Radiology, University of Pennsylvania, Philadelphia, Pa (M.S.); Department of Radiology and Imaging Sciences, Emory University, Atlanta, Ga (C.C.M.); Department of Radiology, Hospital Clínic, Barcelona, Spain (L.O., L.D.); Department of Radiology, Northwell Health, Manhasset, NY (C.G.F.); Department of Radiology, University of Cincinnati, Cincinnati, Ohio (M.M.); and Department of Vascular and Interventional Radiology, Singapore General Hospital, Singapore (B.S.T.)
| | - Carolyn C. Meltzer
- From the Department of Radiology, University of Washington, 1959 NE Pacific St, Seattle, WA 98195 (M.M.B.); Department of Radiology, University of Pennsylvania, Philadelphia, Pa (M.S.); Department of Radiology and Imaging Sciences, Emory University, Atlanta, Ga (C.C.M.); Department of Radiology, Hospital Clínic, Barcelona, Spain (L.O., L.D.); Department of Radiology, Northwell Health, Manhasset, NY (C.G.F.); Department of Radiology, University of Cincinnati, Cincinnati, Ohio (M.M.); and Department of Vascular and Interventional Radiology, Singapore General Hospital, Singapore (B.S.T.)
| | - Laura Oleaga
- From the Department of Radiology, University of Washington, 1959 NE Pacific St, Seattle, WA 98195 (M.M.B.); Department of Radiology, University of Pennsylvania, Philadelphia, Pa (M.S.); Department of Radiology and Imaging Sciences, Emory University, Atlanta, Ga (C.C.M.); Department of Radiology, Hospital Clínic, Barcelona, Spain (L.O., L.D.); Department of Radiology, Northwell Health, Manhasset, NY (C.G.F.); Department of Radiology, University of Cincinnati, Cincinnati, Ohio (M.M.); and Department of Vascular and Interventional Radiology, Singapore General Hospital, Singapore (B.S.T.)
| | - Christopher G. Filippi
- From the Department of Radiology, University of Washington, 1959 NE Pacific St, Seattle, WA 98195 (M.M.B.); Department of Radiology, University of Pennsylvania, Philadelphia, Pa (M.S.); Department of Radiology and Imaging Sciences, Emory University, Atlanta, Ga (C.C.M.); Department of Radiology, Hospital Clínic, Barcelona, Spain (L.O., L.D.); Department of Radiology, Northwell Health, Manhasset, NY (C.G.F.); Department of Radiology, University of Cincinnati, Cincinnati, Ohio (M.M.); and Department of Vascular and Interventional Radiology, Singapore General Hospital, Singapore (B.S.T.)
| | - Mary Mahoney
- From the Department of Radiology, University of Washington, 1959 NE Pacific St, Seattle, WA 98195 (M.M.B.); Department of Radiology, University of Pennsylvania, Philadelphia, Pa (M.S.); Department of Radiology and Imaging Sciences, Emory University, Atlanta, Ga (C.C.M.); Department of Radiology, Hospital Clínic, Barcelona, Spain (L.O., L.D.); Department of Radiology, Northwell Health, Manhasset, NY (C.G.F.); Department of Radiology, University of Cincinnati, Cincinnati, Ohio (M.M.); and Department of Vascular and Interventional Radiology, Singapore General Hospital, Singapore (B.S.T.)
| | - Lluis Donoso-Bach
- From the Department of Radiology, University of Washington, 1959 NE Pacific St, Seattle, WA 98195 (M.M.B.); Department of Radiology, University of Pennsylvania, Philadelphia, Pa (M.S.); Department of Radiology and Imaging Sciences, Emory University, Atlanta, Ga (C.C.M.); Department of Radiology, Hospital Clínic, Barcelona, Spain (L.O., L.D.); Department of Radiology, Northwell Health, Manhasset, NY (C.G.F.); Department of Radiology, University of Cincinnati, Cincinnati, Ohio (M.M.); and Department of Vascular and Interventional Radiology, Singapore General Hospital, Singapore (B.S.T.)
| | - Bien Soo Tan
- From the Department of Radiology, University of Washington, 1959 NE Pacific St, Seattle, WA 98195 (M.M.B.); Department of Radiology, University of Pennsylvania, Philadelphia, Pa (M.S.); Department of Radiology and Imaging Sciences, Emory University, Atlanta, Ga (C.C.M.); Department of Radiology, Hospital Clínic, Barcelona, Spain (L.O., L.D.); Department of Radiology, Northwell Health, Manhasset, NY (C.G.F.); Department of Radiology, University of Cincinnati, Cincinnati, Ohio (M.M.); and Department of Vascular and Interventional Radiology, Singapore General Hospital, Singapore (B.S.T.)
| | - for the RSNA COVID-19 Task Force
- From the Department of Radiology, University of Washington, 1959 NE Pacific St, Seattle, WA 98195 (M.M.B.); Department of Radiology, University of Pennsylvania, Philadelphia, Pa (M.S.); Department of Radiology and Imaging Sciences, Emory University, Atlanta, Ga (C.C.M.); Department of Radiology, Hospital Clínic, Barcelona, Spain (L.O., L.D.); Department of Radiology, Northwell Health, Manhasset, NY (C.G.F.); Department of Radiology, University of Cincinnati, Cincinnati, Ohio (M.M.); and Department of Vascular and Interventional Radiology, Singapore General Hospital, Singapore (B.S.T.)
| |
Collapse
|
5
|
Kruskal JB, Rosen MP, Hara AK, Canon C, Wald C. Enabling Your Radiology Business to Thrive Strategic Lessons Learned During the Initial and Subsequent Surges of the Covid-19 Pandemic. Acad Radiol 2021; 28:393-401. [PMID: 33455861 DOI: 10.1016/j.acra.2021.01.005] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2020] [Revised: 01/06/2021] [Accepted: 01/06/2021] [Indexed: 01/07/2023]
Abstract
The Covid-19 pandemic surges of 2020 resulted in major operational, personal, and financial impacts on US radiology practices. In response, a series of strategic and intentional operational changes were implemented, varying by practice size, structure and model. In reviewing the many business lessons that we learned during the pandemic, it became clear that for a business to be successful, a host of additional supportive factors are necessary. In addition to timely expense reductions, optimizing revenue capture and close monitoring and management of cash and reserves available for use, we also consider effective leadership and communication strategies, maintenance of a healthy and adequately staffed team, support for a remote work environment and flexible staffing models. Other ingredients include effectively embracing digital media for communications, careful attention to current and new stakeholders and the service delivered to them, understanding federal and state regulatory changes issued in response to the pandemic, close collaboration with the Human Resources office, and an early focus on redesigning your future practice structure and function, including disaster and downtime planning. This review aims to share lessons to enable leaders of an imaging enterprise to be better prepared for similar and future surges.
Collapse
Affiliation(s)
- Jonathan B Kruskal
- Department of Radiology, Beth Israel Deaconess Medical Center, One Deaconess Road, Boston, MA 02215.
| | - Max P Rosen
- Department of Radiology, University of Massachusetts Medical School, Worcester, Massachusetts
| | - Amy K Hara
- Department of Radiology, Mayo Clinic, Phoenix, Arizona
| | - Cheri Canon
- Department of Radiology, The University of Alabama at Birmingham, Birmingham, Alabama
| | - Christoph Wald
- Department of Radiology, Lahey Hospital and Medical Center, Burlington, Massachusetts
| |
Collapse
|
6
|
Vagal A, Mahoney M, Anderson JL, Allen B, Hudepohl J, Chadalavada S, Choe KA, Kapur S, Gaskill-Shipley M, Makramalla A, Brown A, Braley S, England E, Scheler J, Udstuen G, Rybicki FJ. Recover Wisely From COVID-19: Responsible Resumption of Nonurgent Radiology Services. Acad Radiol 2020; 27:1343-1352. [PMID: 32933802 PMCID: PMC7416732 DOI: 10.1016/j.acra.2020.08.002] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2020] [Revised: 08/04/2020] [Accepted: 08/04/2020] [Indexed: 01/17/2023]
Abstract
Rationale and Objectives Following state and institutional guidelines, our Radiology department launched the “Recover Wisely” for all nonurgent radiology care on May 4, 2020. Our objective is to report our practice implementation and experience of COVID-19 recovery during the resumption of routine imaging at a tertiary academic medical center. Materials and Methods We used the SQUIRE 2.0 guidelines for this practice implementation. Recover Wisely focused on a data driven, strategic rescheduling and redesigning patient flow process. We used scheduling simulations and meticulous monitoring and control of outpatient medical imaging volumes to achieve a linear restoration to our pre-COVID imaging studies. We had a tiered plan to address the backlog of rescheduled patients with gradual opening of our imaging facilities, while maintaining broad communication with our patients and referring clinicians. Results Recover Wisely followed our anticipated linear modeling. Considering the last 10 weeks in the recovery, outpatient growth was linear with an increase of approximately 172 cases per week, (R2 =0.97). We achieved an overall recovery of 102% in week 10, as compared to average weekly pre-COVID outpatient volumes. The modalities recovered as follows in outpatient volumes: CT (113%), MRI (101%), nuclear medicine including PET (138%), mammograms (97%), ultrasound (99%) and interventional radiology (106%). When compared to identical 2019 calendar weeks (May 4, 2020–July 10, 2020), the total 2020 radiology volume was 11% reduced from the 2019 volume. The reduction in total weighted relative value units was 8% in this time period, as compared to 2019. Conclusion Our department utilized a data-driven, team approach based on our guiding principles to “Recover Wisely.” We created and implemented a methodology that achieved a linear increase in outpatient studies over a 10-week recovery period.
Collapse
|