Gemini-2 Effectiveness in Recommending Cardiovascular Imaging Scans and Protocols

Background: Reviewing imaging referrals is an important yet time-consuming part of the radiology workflow. Radiologists assign the appropriate scan modality and imaging protocol, relating to the clinical question. Integrating large language models (LLM) into this process may potentially improve efficiency.

Aim: To evaluate the effectiveness of LLMs in recommending imaging modality and protocols for common cardiovascular indications based on clinical questions.

Methods: We retrospectively analyzed the referral notes for consecutive cardiac CT and MRI scans performed in a single tertiary center between January and March 2024. Overall, 200 cases were included. The clinical referral notes were entered into Gemini-2.0-Flash and Gemini-2.0-mini models, requesting recommendations on the most suitable imaging modality and protocol. Two independent board-certified radiologists blindly reviewed the referrals and recommended the imaging modality and protocols accordingly. The chatbot’s protocol recommendations were compared with the radiologists’ recommendations and the actual imaging protocols applied. Agreement between readers was assessed using Cohen’s kappa coefficient to account for agreement beyond chance and to assess the reliability of protocol selection.

Results: Gemini-2.0-Flash and Gemini-2.0-mini matched radiologists in 75%–77% of cases (percent agreement). Radiologists matched each other in 76.5% to 95%. Cohen’s Kappa analysis showed substantial agreement between LLM and experts (κ = 0.705–0.731), which was comparable to inter-expert substantial agreement (κ = 0.733–0.94).

Conclusions: Gemini-2.0-Flash and Gemini-2.0-mini performance was within the range of radiologist’s functionality for cardiovascular imaging protocol selection. Prompt refinement further improved agreement, highlighting the importance of iterative human-in-the-loop in order to further enhance and optimize automatic LLM-based protocol selection.