Application of Vision-Language Models in the Automatic Recognition of Bone Tumors on Radiographs: A Retrospective Study

Abstract

Background: Vision-language models show promise in medical image interpretation, but their performance in musculoskeletal tumor diagnostics remains underexplored. Objective: To evaluate the diagnostic accuracy of six large language models on orthopedic radiographs for tumor detection, classification, anatomical localization, and X-ray view interpretation, and to assess the impact of demographic context and self-reported certainty. Methods: We retrospectively evaluated six VLMs on 3746 expert-annotated orthopedic radiographs from the Bone Tumor X-ray Radiograph dataset. Each image was analyzed by all models with and without patient age and sex using a standardized prompting scheme across four predefined tasks. Results: Over 48,000 predictions were analyzed. Tumor detection accuracy ranged from 59.9–73.5%, with the Gemini Ensemble achieving the highest F1 score (0.723) and recall (0.822). Benign/malignant classification reached up to 85.2% accuracy; tumor type identification 24.6–55.7%; body region identification 97.4%; and view classification 82.8%. Demographic data improved tumor detection accuracy (+1.8%, p < 0.001) but had no significant effect on other tasks. Certainty scores were weakly correlated with correctness, with Gemini Pro highest (r = 0.089). Conclusion: VLMs show strong potential for basic musculoskeletal radiograph interpretation without task-specific training but remain less accurate than specialized deep learning models for complex classification. Limited calibration, interpretability, and contextual reasoning must be addressed before clinical use. This is the first systematic assessment of image-based diagnosis and self-assessment in LLMs using a real-world radiology dataset.

Ähnliche Arbeiten