A human in the loop evaluation of generative artificial intelligence for translating medicine instructions into low resourced South African languages

Abstract

South Africa is home to a rich tapestry of cultural and linguistic diversity, with 11 official spoken languages and South African Sign Language recognised as the twelfth. While this plurality reflects the nation’s vibrant social fabric, it poses challenges for equitable healthcare delivery and pharmacy practice. The predominance of English as the primary language used on medication labels, patient information leaflets and in healthcare communication can create language barriers. This can limit patient understanding and potentially compromise safe and effective medicine use for non-English-speaking populations. This study examines the development and implementation of a generative Artificial Intelligence translation workflow. Developed with OpenAI’s ChatGPT Builder and configured by the research team, the translation application has the potential to translate medicine instructions at pharmacy dispensaries into four major South African languages. Two scoring rounds evaluated the accuracy of translated medicine information. In round one, Section two of a South African Health Products Regulatory Authority-approved Patient Information Leaflet was translated into Afrikaans, isiXhosa, isiZulu, and siSwati using a customised General-purpose Technology. Linguists assessed outputs with a structured scoring tool measuring accuracy, fluency, and cultural appropriateness. After model refinement, round two repeated evaluation on a comparable selection from a new Section two South African Health Products Regulatory Authority-approved Patient Information Leaflet to reduce bias from prior feedback. Across two evaluation rounds, translation accuracy improved for siSwati and isiZulu, with substantial reductions in critical error rates (siSwati: RR = 0.57; 95% CI 0.38–0.86; isiZulu: RR = 0.58; 95% CI 0.43–0.79). These findings indicate a 40–45% improvement in translation precision following model refinement. In contrast, isiXhosa demonstrated a marked deterioration (RR = 10.34; 95% CI 7.34–14.57), while Afrikaans remained unchanged due to an absence of critical errors. Overall, the iterative retrieval-augmented and human-in-the-loop design improved translation quality for selected languages but revealed language-specific disparities in model performance. A human‑in‑the‑loop, retrieval‑augmented approach can strengthen General-purpose Technology‑based medicine translations. Future work will build human‑verified datasets to improve accuracy and reliability across South African languages.

Ähnliche Arbeiten