LBP3 Comparative evaluation of ChatGPT and google bard as artificial intelligence-based tools together with expert validation in SLE patient’s education

Abstract

<h3>Objectives</h3> 1) Compare and assess the accuracy of Google Bard and ChatGPT in SLE patients education. 2) Comparison, Scoring, and Validation of the answers by AI tools from rheumatologists vs other specialities treating lupus. <h3>Methods</h3> The prospective study was performed in the Dept. of Medicine, KMC Manipal, from November 2023 to January 2024. Twenty commonly asked questions by SLE patients (<b>table 1</b>) were retrieved by ChatGPT and Google Bard, which were then evaluated by expert consultants. This procedure was iterated thrice. SMOG (<b>Simple Measure of Gobbledygook)</b> scoring was done to check the clarity and accessibility of each question, and the best was chosen based on the highest readability scoring. 17 experts treating lupus patients from different domains were engaged to evaluate these answers through an online Google form. Their role was to access and score these answers derived from the above AI tools on a scale from 0 to 10, with grades ranging from 10 to highly satisfactory to less than 5: not satisfactory. Informed consent and approval were obtained from all the experts. <h3>Results</h3> Google Bard showed more relevant information with comprehensiveness and clarity on most of the questions than ChatGPT (<b>figure 1).</b> There were a few questions for where we received the comments as follows: ChatGPT expert’s comment: Answer 1: A very concise answer, not descriptive common characteristics are not highlighted. Answer 7: missing many markers mentioned rare ones. Answer 14: incorporated more clinical symptoms rather than indices to differentiate. Missed CRP and other markers. Answer 19: Has not recognized the importance of other specialities in treating SLE as it’s a multi-organ disease that requires a multi-disciplinary approach. Google Bard’s Answer 2: has not gone into intrinsic details of the etiopathogenesis. Answer 5: not a clear explanation and less clear. Rheumatologists rely more on Google Bard, whereas ChatGPT was found to be best among other specialities treating lupus (<b>figure 2 a,b</b>). <h3>Conclusions</h3> Our study revealed Google Bard is significantly more precise and comprehensive about the SLE than ChatGPT. SLE is a heterogenous disease that involves multi-specialties along with a rheumatologist as an expert to treat and make critical decisions with organ involvement. SMOG scoring ultimately contributes to the successful transmission of information and knowledge across diverse audiences in an easily readable manner. <h3>Acknowledgements</h3> The authors did not receive any funding for the study.

Ähnliche Arbeiten