Comparison of chatbots’ accuracy in endodontics questions in dentistry specialization exam in Türkiye: ChatGPT-4o, Gemini Advanced, Copilot, and Claude

Abstract

The aim of this study was to evaluate the performance of various chatbots (ChatGPT-4o, Gemini Advanced, Microsoft Copilot, and Claude) on endodontic questions in the Turkish Dentistry Specialization Exam (DUS) based on topic, year, and Bloom’s taxonomy. Additionally, the study aimed to contribute to the use of artificial intelligence chatbots as a supplementary learning tool by evaluating their reliability and limitations in dental education. The accuracy rates of ChatGPT-4o, Gemini Advanced, Microsoft Copilot, and Claude were assessed using 122 endodontic questions from the DUS between 2012 and 2021. Each question was given to the chatbots in a new chat. Accuracy rates were compared based on year, topic, and Bloom’s taxonomy. The accuracy rates of the chatbots were similar: 82.8% for ChatGPT-4o, 83.6% for Gemini Advanced, 77.9% for Copilot, and 82.8% for Claude. Additionally, the accuracy of Copilot and Claude was significantly higher during 2012–2015 compared to 2016–2021 (p = 0.020; p = 0.018). According to Bloom’s taxonomy, ChatGPT-4o, Copilot, and Claude showed higher accuracy in 2012–2015 than in 2016–2021 on low-level questions (p = 0.040; p = 0.011; p = 0.005). Within the limitations of this study, chatbots showed similar overall accuracy and performed better on earlier exams.

Ähnliche Arbeiten