Estimating the Prevalence of Generative AI Use in Medical School Application Essays: A Cross-Sectional Study (Preprint)

Abstract

<sec> <title>BACKGROUND</title> Generative artificial intelligence (AI) tools became widely available to the public in November 2022. The extent to which these tools are being used by medical school applicants during the admissions process is unknown. </sec> <sec> <title>OBJECTIVE</title> We aimed to estimate the extent of generative AI use in cohorts of applicants spanning the rollout of these tools. </sec> <sec> <title>METHODS</title> We retrospectively analyzed 6,000 essays submitted to a U.S. medical school in 2021–2022 (baseline, before wide availability of AI) and in 2023–2024 (test year) to estimate the prevalence of AI use and its relation to other application data. We used GPTZero, a commercially available detection tool, to generate a metric (P_human) for the likelihood that each essay was human-generated, ranging from 0 (entirely AI) to 1 (entirely human). </sec> <sec> <title>RESULTS</title> Fully human-generated negative controls demonstrated a median P_human of 0.93, while AI-generated positive controls demonstrated a median P_human of 0.01. Personal Comments essays submitted in the 2023–2024 cycle had a median P_human of 0.77 (95% confidence interval 0.76–0.78), versus 0.83 (95% CI 0.82–0.85) during the 2021–2022 cycle. Approximately 12.3 and 2.7% of essays were evaluated as having P_human &lt; 0.5 in the test and baseline year, respectively. Secondary essays demonstrated lower P_human than AMCAS Personal Comments essays, suggesting more AI use. In multivariate analysis, younger age and higher GPA were significantly associated with lower P_human. No differences were observed in gender, MCAT score, undergraduate major, or socioeconomic status. P_human was not predictive of admissions outcomes in uni- or multivariate analyses. </sec> <sec> <title>CONCLUSIONS</title> An AI detection algorithm identified significantly increased use of generative AI in 2023–2024 medical school admission applications, as compared to the 2021–2022 baseline. AI use did not appear to confer an admissions advantage. While these results provide information about the applicant pool as a whole, AI detection is imperfect. We do not recommend deploying AI detection on individual applications in live admissions cycles. </sec> <sec> <title>CLINICALTRIAL</title> Trial Registration: None </sec>

Ähnliche Arbeiten