Differential Accuracy and Concordance of ChatGPT and DeepSeek for Image-Based Questions in Undergraduate Medical Education

AI Summary

ChatGPT outperformed DeepSeek overall (72.9% vs 65.1%), with moderate inter-model agreement (κ = 0.591).
Inclusion of images did not increase accuracy; image-only performance declined, indicating limited benefit from isolated visual information.
Models showed modality-specific strengths, but overall remained text-driven clinical reasoners rather than effective multimodal diagnostic systems.

Med Sci Educ. 2026 Aug 4;36(6):3009-3017. doi: 10.1007/s40670-026-02839-9. eCollection 2026 Dec.

ABSTRACT

Large language models (LLMs) such as ChatGPT (GPT) and DeepSeek (DS) are increasingly explored in medical education and practice. This study compared the accuracy and concordance of GPT and DS in answering multiple-choice Progress Test questions (2013-2024) across three input formats: text-only, image-only, and full-question (vignette + image). GPT achieved 72.9% overall accuracy versus 65.1% for DS, with moderate inter-model agreement (κ = 0.591). Critically, for both models, full-question accuracy did not exceed text-only accuracy, indicating that the addition of visual content provided no global incremental benefit. Conversely, accuracy dropped significantly in the image-only condition, confirming that isolated visual information was less informative than textual history. Category-specific analyses revealed relative strengths for GPT in skin and body pictures and for DS in microscopic and radiographic images, though these did not translate into an overall multimodal advantage. Questions answered correctly had lower difficulty indices, while discrimination indices did not differ between correct and incorrect responses, and performance was similar across cognitive levels. These findings demonstrate that, under the tested conditions, LLM performance was predominantly text-driven rather than visuospatially integrated. Accordingly, ChatGPT and DeepSeek function as proficient text-based clinical reasoners, but not as effective multimodal diagnostic systems in this educational assessment context.

SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at https://doi.org/10.1007/s40670-026-02839-9.

PMID:43031833 | PMC:PMC13759941 | DOI:10.1007/s40670-026-02839-9

Document this CPD