A Comparative Evaluation of Three Artificial Intelligence Chatbots' Responses to the Most Commonly Searched Inquiries on Urethral Stricture Disease: Cross-Sectional Analytical Study

Abstract

Objective: This study aimed to evaluate and compare the performance of 3 artificial intelligence (AI) models -ChatGPT, Gemini 2.0 AI, and Lama AI- in answering frequently asked questions about urethral stricture disease (USD), using the European Association of Urology guidelines as a reference. The models' responses were assessed in terms of scientific accuracy, compliance with guidelines, and accessibility. Material and Methods: Trends related to USD were analyzed over the last 5 years using Google Trends and SEMrush. Based on this analysis, 25 commonly searched questions were selected and grouped into 4 domains: ''definition and etiology'', ''clinical assessment'', ''treatment methods and future perspectives'', and ''multidisciplinary management and myths''. These standardized questions were posed to each AI model. The responses were evaluated for scientific content using Ensuring Quality Information for Patients and Modified DISCERN criteria, readability using the Flesch Reading Ease Score (FRES), and textual complexity using the Flesch-Kincaid Grade Level (FKGL). Results: ChatGPT and Gemini scored significantly higher than Lama in scientific accuracy across groups 1, 2, and 4 (p<0.001). In Modified DISCERN, ChatGPT and Gemini scored between 2.60-3.00, while Lama performed better in certain domains, particularly in treatment-related responses. FRES and FKGL analyses revealed that ChatGPT and Gemini provided more readable content than Lama, with Gemini achieving the best readability scores. Conclusion: ChatGPT and Gemini outperformed Lama AI in delivering accurate, guideline-compliant, and readable information on USD. Nevertheless, variations in accuracy highlight the importance of medical expert supervision when using AI tools in clinical contexts.

Ähnliche Arbeiten