A Good College Essay but a Bad Police Report: A Triple-Blind Expert Evaluation of AI-Assisted Police Reporting

Abstract

Police departments are adopting AI-powered report-writing tools, with vendors claiming better-quality reports. We test that claim in a triple-blind experiment: 92 expert law-enforcement reviewers (sergeants and above with average of 21.8 years of professional responsibility for report approval) produced 354 rater-report evaluations of 80 reports drawn from a randomized controlled trial, without knowing which reports used AI.Computational text analysis confirms AI reports are more complex, less readable, and written at a higher grade level. Reviewers find no composite-level difference but rate AI reports substantively and significantly lower on accuracy (𝑝 = 0.038), with all six dimension-specific estimates directionally negative (sign-test 𝑝 = 0.031); and approval rates are identical across conditions (~22%, matching prior field estimates of first-attempt approval). Reviewers cannot distinguish AI-assisted from conventional reports, with accuracy rates equal to a coin flip (AUC = 0.50). Supervisor ratings do not track objective complexity on either Flesch Reading Ease or a separate letter-frequency Scrabble measure; and supervisors rate AI reports as less complete on a dimension where the tool’s transcript-only architecture ensures incompleteness, yet approve them anyway.Detection is not what supervisors are asked to do. The substantive oversight failures are that their quality rubrics do not weight the complexity signal AI produces, and do not catch the information gap the transcript architecture produces. Governance frameworks assuming supervisors can detect AI authorship need structural substitutes; frameworks assuming supervisors catch the quality consequences need retraining on what “quality” means for a document read by non-specialists.

Ähnliche Arbeiten