Assessing ChatGPT vs. evidence-based online responses for polycystic ovary syndrome self-management and education: an international cross-sectional blinded survey of healthcare professionals

Abstract

Artificial intelligence (AI)-powered large language models, such as ChatGPT, are increasingly used by the public for health information. The reliability of such novel AI-tools in providing credible polycystic ovary syndrome (PCOS) information/advice requires investigation. Healthcare professionals involved in PCOS care ( n = 43 from 14 countries) used a 5-point Likert scale to evaluate ChatGPT-generated responses to frequently asked questions about PCOS against the corresponding patient-orientated, evidence-based recommendations/responses available online. ChatGPT responses were rated significantly higher than the evidence-based responses for 11 of the 12 study questions, with moderate to large effect sizes ( r r b = −0.46 to −1.00; all p -values <0.05), with ChatGPT answers being rated on average 0.824 units higher. Scoring agreement varied (poor to fair), with seven questions showing statistically fair agreement (κ = 0.24–0.37, p < 0.05). Readability analyses found no statistically significant differences between ChatGPT and evidence-based responses. However, using ChatGPT for simplifying the responses resulted in significant improvement. ChatGPT holds potential as a complementary patient self-education tool in PCOS, capable of interactive engagement and simplifying medical language. Further research is needed to identity optimal integration of AI tools and validate their clinical applicability for PCOS self-education/management.

Ähnliche Arbeiten