Cautionary note: Chatbots generate fictitious references on the topics related to oral pathology

Abstract

While chatbots such as Chat Generative Pre-Trained Transformer (ChatGPT) and BARD have gained significant popularity in aiding researchers in crafting scientific articles, concerns persist, particularly regarding their accuracy in generating references. This is especially pronounced in specialized fields like oral pathology, where precision is paramount. We conducted a comprehensive comparison between ChatGPT and BARD in the context of generating references for five frequently researched topics within oral pathology. To assess accuracy, we subjected fifty references generated by each model to rigorous evaluation based on multiple parameters. The validation process involved meticulous cross-referencing across prominent scientific databases. ChatGPT demonstrated a reference accuracy of only 10 %, revealing notable inconsistencies across various parameters. Conversely, BARD exhibited a substantial 88 % inaccuracy rate in total references, with only 4 references managing to provide the correct title. Notably, the DOI led to the accurate manuscript in only 12 % (9) of references generated by ChatGPT and 4 % (8) of references from BARD. Analysis indicated significantly better performance by the ChatGPT in comparison to BARD (p < 0.05). Prudence is advised when incorporating references generated by ChatGPT and BARD on topics related to oral pathology. There is a pressing need for publishers to fortify systems for scrutinizing references and integrate reputable scientific databases into Chatbot systems to ensure accuracy.

Ähnliche Arbeiten