Physician evaluation of artificial intelligence generated educational materials compared with hospital website resources for transcatheter aortic valve replacement

Abstract

As transcatheter aortic valve replacement (TAVR) becomes increasingly common, patients are seeking procedural information from Large Language Models (LLMs; AI systems trained on large text datasets to generate human-like responses) like ChatGPT. While high-quality patient-facing information is a core component of health care quality, a significant educational gap exists between complex clinical resources and patient health literacy. The efficacy of artificial intelligence (AI) as a pedagogical tool to bridge this gap, compared with traditional institutionally produced hospital materials, remains largely unexplored in procedural cardiology. We conducted a cross-sectional comparative evaluation study in which fifteen physicians with experience in transcatheter aortic valve replacement assessed responses to eight frequently asked procedural questions. Questions were derived from seven major United States hospital websites. For each question, three hospital-generated responses and one artificial intelligence–generated response were evaluated. Responses were rated for scientific accuracy, ease of understanding, and overall satisfaction using a five-point Likert scale. Physicians were also asked to identify which responses were generated by artificial intelligence. The study evaluated AI’s performance as a ‘digital educator,’ focusing on its ability to synthesize complex procedural data into understandable instructional content. Artificial intelligence–generated responses were correctly identified in 45.8% of cases. More frequent use of ChatGPT was associated with higher identification accuracy. Overall, artificial intelligence–generated responses received higher mean ratings for scientific accuracy (3.88 vs. 3.44), ease of understanding (3.78 vs. 3.54), and overall satisfaction (3.77 vs. 3.33) compared with hospital website content. Performance advantages were most evident for procedural definitions, explanations of the procedure, post-procedure expectations, and discussion of potential benefits. Our study suggests that AI-generated education for TAVR can achieve ratings comparable to—and in selected domains higher than—educational materials from major hospital websites, representing an initial exploratory step in evaluating LLM-generated medical content. However, these findings should be interpreted cautiously due to the limited and homogeneous expert sample and the absence of patient-centered evaluation.

Ähnliche Arbeiten