Peer Review in Diagnostic Imaging: Modality-Adjusted Analysis of Disagreement and Reviewer Variability

Background:
Peer review in diagnostic imaging is an important quality improvement tool. However, disagreement rates may reflect imaging modality, case mix, and reviewer scoring behavior, rather than only the quality of the original interpretation. We analyzed a multi-year dataset to characterize disagreement patterns and identify outliers after modality adjustment.

Methods:
We retrospectively included all available imaging peer-review records documented in our institutional database from April 2023 through July 2026. Records without a valid score were excluded from score-based analyses. Score 1 indicated full agreement, scores 2–4 any disagreement, and scores 3–4 significant or major disagreement. Analyses were performed by modality, original interpreting physician, and peer reviewer. For scores 3–4, observed-to-expected (O/E) ratios were calculated using modality-specific baseline rates. Approximate 95% confidence intervals were calculated using a log-normal approximation to the Poisson distribution. Multiple comparisons were addressed using false discovery rate correction.

Results:
Of 11,019 records, 11,018 had valid scores. Full agreement occurred in 8,906 cases (80.8%), any disagreement in 2,112 (19.2%), and significant or major disagreement in 703 (6.4%). Rates of scores 3–4 differed by modality: CT 18.0%, bone densitometry 7.6%, MRI 3.7%, and mammography 3.3% (χ²=540.21, p<0.0001). Reviewer variability was also significant (χ²=1289.45, p<0.0001). After modality adjustment, several reviewers had significantly higher or lower than expected rates. Some crude physician-level outliers were attenuated after adjustment.

Conclusions:
Imaging peer-review data should be interpreted after adjustment for modality and consideration of reviewer behavior. Crude disagreement rates may be misleading, particularly in CT. O/E analysis with FDR correction may improve identification of meaningful quality signals, support reviewer calibration, and guide targeted case review.