Can People Tell the Difference Between AI-Generated Mental Health Vignettes? An Exploratory Comparison of User Evaluations

Abstract

Vignettes are brief, descriptive, hypothetical scenarios that have been used to extract attitudes, beliefs, or perceptions from participants across psychology, healthcare, and human–computer interaction. Traditional vignette development is often time and labor-intensive and large language models (LLMs) like ChatGPT-4o may streamline this process. This exploratory between-subjects online survey (n = 66) compared participants’ perceptions of clinically reviewed LLM-generated versus human-written mental health vignettes describing social anxiety, depression, or schizophrenia. Participants rated each vignette on realism, clarity, engagement, emotional impact, perceived likelihood of AI authorship, and likelihood that the target diagnosis applied. Mixed-effects linear regression analyses showed no statistically significant differences between AI-generated and human-written vignettes for any perceived quality rating; estimated source effects were small (|β| ≤ 0.10) with 95% confidence intervals spanning zero across outcomes. Perceived AI authorship likelihood (β = 0.09, 95% CI [−0.22, 0.40]) and correct-diagnosis likelihood ratings (β = −0.07, 95% CI [−0.30, 0.16]) also did not differ by source. Overall, we did not detect statistically significant differences between AI-generated and human-written vignettes. These findings reflect perceptions of AI-generated vignettes that underwent expert clinical review and suggest that LLMs may assist in vignette generation with expert oversight, while highlighting the need for further research on clinical accuracy, diagnostic validity, and generalizability.

Ähnliche Arbeiten