Feasibility of Using LLMs to Automate Analysis of AI/ML Medical Device Approvals

Abstract

The integration of machine learning (ML) algorithms into medical devices for clinical decision-making is rapidly expanding. However, the specific functionalities and clinical use of ML within these devices often remain unclear, posing potential safety concerns. While manual analysis of ML-driven devices approved by the FDA provides insights, it is inefficient. This study explores the feasibility of using large language models (LLMs) to automate such analyses. We evaluate LLMs based on architecture, training strategies, parameter size, computational demands, and output quality to extract general device characteristics, ML-specific details (e.g., functions, inputs/outputs), and clinical applications (e.g., users, conditions). Analyzing 108 ML device approvals, we found that decoder LLMs excel at extracting explicit information but are computationally intensive, whereas encoder models are more efficient and better at inferring clinical context. Despite these strengths, all LLMs require domain-specific optimization to effectively address ML-related details. These findings serve as benchmark of LLMs in this domain.

Ähnliche Arbeiten