An AI-Assisted Adaptive Boolean Rubric for exercise prescription evaluation: A pilot validation study

Abstract

• Novel AI-assisted framework: Adaptive-PBR enhances exercise prescription evaluation. • Superior Reliability: Boolean rubric (ICC = 0.83) outperforms traditional Likert scales (ICC = 0.65). • High Efficiency: AI-curated rubrics reduce expert evaluation time by 63% with equal precision. • Reduced Bias: Structured Boolean criteria eliminate experience-related scoring bias. • Ecological Validity: Dual-run AI protocol ensures reproducibility in real-world clinical settings. The quality assessment of personalized exercise prescriptions is currently hampered by the subjectivity and inefficiency of traditional rating scales. Artificial intelligence (AI) presents a transformative opportunity for objective, scalable evaluation. This pilot study aimed to develop an AI-assisted evaluation framework and assess the feasibility, reliability, and efficiency of its core component, the Adaptive Precision Boolean Rubric (Adaptive-PBR). Based on ACSM guidelines, we developed a 50-item Precision Boolean Rubric (PBR) and a 10-item Likert scale. To maximize ecological validity, we utilized GPT-4 (via ChatGPT Plus Web Interface) with a dual-run consistency protocol to generate case-specific, 20-item Adaptive-PBRs. Twelve experts evaluated five diverse clinical cases (yielding 180 rating points) using all three instruments under randomized conditions. The Adaptive-PBR demonstrated excellent inter-rater reliability (ICC = 0.83), significantly outperforming the Likert scale (ICC = 0.65) and matching the full PBR (ICC = 0.82). Quantitatively, it achieved high scoring precision (Median 0.78, IQR 0.70–0.85) while reducing evaluation time by approximately 63 % (mean 7.1 vs. 19.5 min) compared to the full PBR. Crucially, the Adaptive-PBR mitigated the subjective variability and experience-related bias observed with the Likert scale. The AI-assisted Adaptive-PBR establishes a feasible, reliable, and efficient evaluation standard. By combining granular criteria with AI-driven adaptability, it offers a robust alternative to subjective scales, with immediate potential as a quality assurance tool in clinical training and practice.

Ähnliche Arbeiten