Evaluating GPT-4’s Semantic Understanding of Obstetric-based Healthcare Text through Nurse Ruth

Abstract

Nurse Ruth, an AI-driven assistant, is designed to support obstetric nursing in resource-limited environments and for non-specialist healthcare providers. To develop and validate Nurse Ruth, we introduced novel evaluation metrics—Semantic Transparency Metric (STM) and Semantic Understanding Metric (SUM)—to assess response accuracy, contextual relevance, and robustness against conventional and adversarial clinical queries. Through iterative refinement and targeted knowledge integration, Nurse Ruth surpassed the 80% threshold for STM and SUM, reinforcing its ability to provide clear, evidence-based, and contextually precise clinical guidance. While excelling in response clarity and contextual accuracy, further improvements are needed to enhance recall in complex, multi-domain obstetric scenarios. A comparative evaluation against leading AI models (GPT-4o, GPT-4, and GPT-o1) for semantic validation demonstrated Nurse Ruth’s superiority. It achieved 100% accuracy on obstetric challenge queries, outperforming general-purpose AI models in both precision and efficiency. Unlike these models, Nurse Ruth delivered concise, rapid responses, making it the most effective system for real-world clinical applications. These findings validate Nurse Ruth’s semantic understanding and establish a replicable framework for AI-driven decision support in specialized medical fields. Future work will focus on refining recall in multi-faceted obstetric cases and validating real-world clinical impact.

Ähnliche Arbeiten