Comparative analysis of large language models and clinicians in thyroid eye disease using structured questionnaires

Abstract

To preliminarily evaluate agreement between large language models (LLMs) and ophthalmic clinicians regarding thyroid eye disease (TED) clinical decision-making, we developed a structured questionnaire for TED-related scenarios. A structured pilot online survey was conducted among 17 experienced oculoplastic and orbital clinicians from eight countries. The questionnaire assessed physicians' approaches to demographics, initial symptom recognition, differential diagnostic reasoning, and early diagnostic decisions in TED. The same questionnaire items were posed to two leading large language models (LLMs) (GPT-5 and Gemini 2.5 Pro). Agreement between LLMs and clinicians was assessed using similarity metrics, including Jaccard index and cosine similarity. Inter-physician agreement was additionally calculated as a reference baseline. The LLMs demonstrated moderate overall agreement with physicians. Sensitivity analyses showed broadly consistent categorical patterns, although confidence intervals were wide. No significant similarity difference was observed among physician's experience levels. GPT showed statistically significant slightly higher alignment with physician responses Gemini across demographic and diagnostic domains (p < 0.05). A strong positive correlation between Jaccard and Cosine similarities for both LLMs (p < 0.001). LLMs exhibit partial alignment with clinician response patterns in a structured, questionnaire-based setting. However, these findings reflect performance under simplified and decontextualized conditions, further studies incorporating real patient data and open-ended clinical tasks are needed.

Ähnliche Arbeiten