Exploring Generative AI Potential in Clinical Decision-Making: Consistency and Accuracy in TAVI versus AVR Recommendations

Background: The selection of Transcatheter Aortic Valve Implantation (TAVI) versus Aortic Valve Replacement (AVR) is a nuanced decision typically made by multidisciplinary Heart Teams, considering patient-specific factors. This study evaluates the consistency and reliability of a custom-configured version of ChatGPT-4o in recommending TAVI or AVR when prompted multiple times with identical patient scenarios.

Methods: A custom GPT-4 was configured by providing it with detailed instructions and the ESC/EACTS guidelines, without additional training or model modification. Patient data (n = 23), including detailed clinical information, were used to query the model three times per patient using the same input prompt. Consistency across the three recommendations (Decisions 1, 2, and 3) was assessed. Sensitivity, specificity, and accuracy were calculated by comparing the model`s recommendations with Heart Team decisions. Inconsistent outputs were analyzed to identify potential sources of variability and model limitations.

Results: The model demonstrated an overall consistency rate of 52.17%, providing identical recommendations across all three prompts for 52.17% of cases. Sensitivity across consistent cases was 55%, with specificity of 55%. Accuracy for consistent predictions was 25%. Inconsistencies in recommendations occurred in 48% of cases, underscoring challenges in model stability. Specificity for inconsistent cases was notably higher at 84.4%, while sensitivity remained at 48%. The custom GPT’s adherence to ESC/EACTS guidelines was generally strong but revealed occasional alignment issues in more complex patient scenarios.

Conclusions: The custom GPT-4 showed moderate consistency, variability in recommendations and limited sensitivity highlighting significant challenges in its clinical applicability. Further model optimization and inclusion of clinical datasets are needed to enhance reliability and clinical decision-making support. This study is still ongoing, and we intend to refine the model further and evaluate its performance on a dataset of around 100 patients to improve its applicability and robustness in clinical decision-making.