Accuracy of Large Language Model Chatbots in Retrieving MRI Safety Conditions for Medical Implants

Background
Retrieval of MRI safety conditions for implanted medical devices is increasingly complex and time-consuming. We evaluated the accuracy and reliability of internet-enabled large language model (LLM) chatbots in retrieving implant-specific MRI safety conditions, using manufacturer documentation as the reference standard.

Methods
Comparative performance evaluation in a purposively constructed cohort of 50 implant device configurations tested across three internet-enabled LLM chatbots. Standardized single-shot prompts requested eight MRI safety parameters. A total of 1,200 parameter-level outputs were independently scored by two readers against manufacturer Instructions for Use and MRI technical manuals using a 5-point ordinal scale; discrepancies were adjudicated. Statistical analyses included ICC(2,1) for interobserver agreement, Friedman test for overall ordinal score comparisons, Wilcoxon signed-rank tests for pairwise comparisons, Wilson 95% confidence intervals for binary accuracy, and two-sided α = 0.05.

Results
Mean ordinal scores were 4.76 for ChatGPT, 4.23 for Gemini, and 4.21 for Claude (χ² = 17.73; P < 0.001). Binary accuracies were 0.94 (95% CI, 0.92–0.96), 0.79 (0.75–0.83), and 0.78 (0.74–0.82), respectively. Interobserver agreement was good (ICC = 0.855; 95% CI, 0.817–0.892). Errors most commonly reflected incomplete parameter extraction, reliance on outdated documentation, or failure to prioritize manufacturer sources.

Conclusions
Internet-enabled LLM chatbots frequently retrieved correct implant-specific MRI safety conditions, with one model outperforming peers, but they remain susceptible to occasional errors. Direct verification against current manufacturer documentation is necessary before clinical use.