Diagnostic Accuracy and Fine-Grained Image-Interpretation Performance of Multimodal Large Language Models on Bilingual Radiology Questions: Cross-Sectional Benchmark Study.
文献摘要
Background Multimodal large language models (LLMs) are rapidly expanding in radiology and digital health; however, their diagnostic performance across diverse clinical scenarios and fine-grained visual reasoning tasks requires systematic evaluation. Objective The study aims to evaluate the diagnostic accuracy and fine-grained visual reasoning performance of 5 multimodal LLMs with bilingual medical imaging questions. Methods This cross-sectional benchmark study evaluated bilingual medical imaging questions using zero-shot prompting on 5 LLMs: GPT-5.2, Gemini 3 Pro, Doubao, Yuanbao, and Llama 4 Maverick. Subgroup analyses by language (English/Chinese), question type (image/text), imaging modality (computed tomography, radiography, magnetic resonance imaging, ultrasound, and nuclear medicine), and anatomical system (neurologic, respiratory, cardiovascular, digestive, genitourinary, and musculoskeletal) evaluated intermodel accuracy and intramodel consistency. Image-based visual reasoning performance was assessed across modality, anatomy, lesion detection, and localization. Results A total of 298 questions (198 image-based, 100 text-based) were evaluated. Gemini 3 Pro achieved the highest overall accuracy (n=268, 89.9%, 95% CI 86.0%-92.9%), significantly outperforming Doubao (n=235, 78.9%; P P P P P P =.02; GPT-5.2, P P =.002; Llama 4 Maverick, P P P =.03), and Yuanbao ( P =.03) favored English questions, while intramodel performance remained stable across modalities (Yuanbao, P =.46; Doubao, P =.21; GPT-5.2, P =.26; Gemini 3 Pro, P =.96; Llama 4 Maverick, P =.45) and anatomical systems (Yuanbao, P =.82; Doubao, P =.52; GPT-5.2, P =.35; Gemini 3 Pro, P =.13; Llama 4 Maverick, P =.97). Despite near-perfect macroscopic visual recognition (modality and anatomy: 99.0%-100.0%), accuracy dropped in complex tasks. Gemini 3 Pro outperformed Yuanbao ( P P =.002), and Llama 4 Maverick ( P P P =.004), and Llama 4 Maverick ( P =.004) in lesion localization (189/198, 95.5%). Conclusions Gemini 3 Pro demonstrated optimal, comprehensive performance and cross-lingual stability. However, current multimodal LLMs face a technical bottleneck in complex visual interpretation-particularly in fine-grained lesion detection and localization-emphasizing the need to prioritize local feature perception in future model development.