A multi-query, multimodal, receiver-augmented solution to extract contemporary cardiology guideline information using large language models

Abstract

Aims: The aim of the current study was to assess the utility of a state-of-the-art large language model (LLM) based on curated, defined clinical practice recommendations to support clinicians in obtaining point-of-care guidelines for individual patient treatment while maintaining transparency. Methods and results: We combined cloud-based and locally run LLMs with versatile open-source tools to form a multi-query, multimodal, retrieval-augmented generation chain that closely reflects European cardiology guidelines in its responses. We compared the performance of this generation chain to other LLMs including GPT-3.5 and GPT-4 on a 306-question multiple-choice exam with questions consisting of short patient vignettes from various cardiology subspecialties, originally written to prepare candidates for the European Exam in Core Cardiology. On the multiple-choice test, our system demonstrated overall accuracy of 73.53%, while GPT-3.5 and GPT-4 had overall accuracies of 44.03 and 62.26%, respectively. Our system outperformed GPT-3.5 and GPT-4 for the following categories of questions: coronary artery disease, arrhythmia, other, valvular heart disease, cardiomyopathies, endocarditis, adult congenital heart disease, pericardial disease, cardio-oncology, pulmonary hypertension, and non-cardiac surgery. For maximum transparency, the system also provided reference quotes for its recommendations. Conclusion: Our system demonstrated superior performance in question-answering tasks on a set of core cardiology questions as compared with contemporary publicly available chat models. The current study illustrates that LLMs can be tailored to provide documented and accountable guideline recommendations towards actual clinical needs while ensuring recommendations are derived from up-to-date, trustable, and traceable documents.

Ähnliche Arbeiten