A Methodology for Developing and Integrating Large Language Models into Electronic Health Records to Support Clinical Workflows

Abstract

The integration of Large Language Models (LLMs) into clinical practice presents both opportunities and challenges, particularly in terms of reliability, safety, and real-world relevance. This thesis establishes a methodology for the development and integration of LLM-based software in Electronic Health Records (EHR) systems to reduce administrative burden, enhance data accessibility, and support clinical reasoning, while ensuring patient safety and maintaining high-quality care. Throughout this thesis, we analyze each step of the process and implement an in-house chatbot integrated into real-world clinical settings, with a focus on bridging technical progress with clinical value. Four primary objectives guide the research: involving clinicians in the development of these tools, adapting LLMs for local medical contexts, assessing the capabilities through reliable evaluations, and integrating LLM-based software in clinical workflows. Key contributions include: internistai-7b-v0.2, a specialized model combining general and medical corpora to achieve state-of-the-art performance; the Glianorex benchmark, which highlights the limitations of multiple-choice testing for clinical reasoning; the MetaMedQA benchmark, designed to assess cognitive capabilities and expose system deficiencies; and a collaborative design process. These contributions led to the development of an entirely in-house chatbot integrated in an EHR with more than 1,000 users and tens of thousands of interactions. Overall, the thesis demonstrates that the integration of LLMs in clinical workflows is feasible, provided that software engineering practices are adapted to the novel challenges presented by LLMs. These adaptations include robust cognitive evaluation, domain-adaptive training, and participatory design. The findings highlight both the promise of LLMs in improving clinical workflows and the bottleneck created by the misalignment of automated evaluations and real-world needs.

Ähnliche Arbeiten