Automation bias and timing of artificial intelligence decision support in screening mammography

Abstract

A reader study was performed using 150 screening mammograms (75 cancers: 75 normal) from the Dutch Screening Program (median age, 61 years [IQR, 55-67 years]). Examinations were analyzed using an FDA-approved commercial artificial intelligence (AI) system. The dataset was enriched with examinations where AI performance was suboptimal, defined as probability-of-malignancy (PoM) scores <43 (out of 100) for cancer cases or PoM≥43 for noncancer cases. Nine Dutch breast cancer screening radiologists read all examinations with (i) no AI support, (ii) on-request AI support (AI examination-level category displayed immediately but AI-detected suspicious region markers displayed only after a button click), and (iii) full concurrent AI support (all AI findings shown immediately). Radiologists recorded their recall decision and PoM score (1-100), and completed a post-study survey. Area under the receiver operating characteristic curve (AUC), sensitivity, and specificity were compared between the three reading conditions and across optimal and suboptimal AI results using analysis of variance (MRMCaov). Reading time was compared using bootstrap resampling. For examinations with optimal AI results, AUC was higher when reading with AI support than when reading without AI support (on-request: 0.967 vs 0.899; P<0.001, full concurrent: 0.970 vs 0.899, P<0.001). Conversely, for examinations with suboptimal AI results, AUC was lower with AI support than without it (on-request: 0.230 vs 0.383; P=0.006, full concurrent: 0.213 vs 0.383, P=0.002). Overall, there was no significant difference in AUC between the two AI-supported reading methods (on-request: 0.830, full concurrent: 0.829; P=0.89). For most reading conditions, sensitivity, specificity, and reading time did not significantly differ. All radiologists preferred to use AI support, and seven preferred it to be on request. This study suggests that reading with AI support can improve radiologist performance when AI results are accurate, but may reduce performance when AI results are suboptimal, independent of when AI region marks are displayed.

Ähnliche Arbeiten