Artificial Intelligence Chatbots Taking American Board of Endodontics Simulated Oral Board Examination

Abstract

INTRODUCTION: The aim of this study was to assess the overall performance of artificial intelligence (AI) chatbots in taking the American Board of Endodontics simulated Oral Board Examination. METHODS: Three oral board cases were constructed by 2 academic board-certified endodontists. Each case included a comprehensive patient profile consisting of medical history, dental history, and results of clinical testing, followed by 20 consecutive open-ended oral board-style questions. Two publicly accessible AI chatbots were selected to take the exam: GPT-4o and Gemini-2.5 Pro. Responses were scored based on a comprehensive rubric on a 4-point ordinal scale (0-3) by the same 2 examiners independently: response validity, citation validity, and overall performance score. A Cumulative Link Mixed Model (proportional odds) was used with fixed effects for chatbot and case, and random intercepts for reviewer and question to analyze and compare the performance of models, that is, inter- and intra-chatbot comparisons. RESULTS: Gemini-2.5 Pro and GPT-4o achieved high mean overall performance scores of 2.83 (±0.42) and 2.73 (±0.51), respectively. Cumulative Link Mixed Model showed no significant difference between the 2 chatbots in probability of receiving an excellent score (ie, 3) in response validity (odds ratio = 2.44, 95% confidence interval [0.98-6.06], P = .054) or in overall performance (odds ratio = 2.04, 95% confidence interval 0.97-4.30, P = .061). There was a positive correlation between response validity and citation validity for GPT-4o (ρ = 0.21, P = .019). CONCLUSIONS: Both chatbots scored considerably high in the simulated American Board of Endodontics Oral Board Examination. Results of this study support the concept of using AI chatbots as aid in endodontic education.

Ähnliche Arbeiten