研究揭示医疗视觉模型按平均准确率评测会掩盖单一病症的完全失效
Aggregate Accuracy Hides Diagnosis-Specific Failure in Medical Vision-Language Models
这篇审计会改变你看医疗模型榜单的方式：两个知名开源模型对气胸全部漏诊，还能流畅描述体征再否认它。做医疗 AI 选型或部署的人必须看，评估任何 VLM 都该按病症拆分指标，别信平均分。
一篇新论文审计了五个开源医疗视觉语言模型：MedGemma-4B、Lingshu-7B、HuatuoGPT-Vision-7B、Qwen2.5-VL-7B 和 MedVLM-R1，使用按性别、年龄、病症和标签平衡的 6048 道 ChestX-ray14 问题。在 432 张气胸阳性 X 光片上，HuatuoGPT-Vision 和 Qwen2.5-VL 检出 0 例，而 MedGemma 和 Lingshu 分别检出 38% 和 67%。两个失效模型对全部 864 道气胸问题一律回答"否"，平衡准确率只有 0.500，等于随机猜测。
失效原因排除了提示方式、图像分辨率、标签歧义和模型规模：同一模型在受控对比下对结节召回率 79%、对气胸只有 6%；放大十倍到 72B 参数后 Qwen2.5-VL 仍只检出 4%；在带放射科医生标注掩膜的独立数据集 SIIM-ACR 上，失效模型对 432 例确诊气胸（含 144 例最大病灶）的检出仍为零。更危险的是，这些模型会先流畅描述气胸的解剖学特征、随后否认它存在，且不作任何保留或拒答。
通过读取模型的 logits 和内部状态，作者证明这不是"看不见"：线性探针能从隐藏表示中恢复气胸信息，说明病症在内部已被表征，只是在输出层被抑制。论文结论是平均准确率无法证明医疗 VLM 能识别某个病症，必须报告按病症拆分的召回率和特异度。
Aggregate Accuracy Hides Diagnosis-Specific Failure in Medical Vision-Language Models
Medical vision-language models (VLMs) are compared mainly by accuracy averaged across findings, a summary that can hide complete failure on a single diagnosis. We audit five open medical VLMs (MedGemma-4B, Lingshu-7B, HuatuoGPT-Vision-7B, Qwen2.5-VL-7B, and MedVLM-R1) on 6,048 ChestX-ray14 questions balanced by sex, age, finding, and label. On the 432 pneumothorax-positive X-rays, HuatuoGPT-Vision and Qwen2.5-VL detect zero cases while MedGemma and Lingshu detect 38% and 67% of the same images. The two failing models answer "no" to all 864 pneumothorax cases (balanced accuracy 0.500, chance), yet produce fluent, anatomically specific explanations that name the signs of a collapsed lung and then deny them, with no hedge or abstention. This failure is not an artifact of prompting, image resolution, label ambiguity, or model scale. It persists across four prompt formats, at native resolution, and on single-label cases. A within-model control shows the same forced prompt gives HuatuoGPT 79% recall on nodule but only 6% on pneumothorax. It survives a tenfold scale increase (Qwen2.5-VL-72B detects 4%) and replicates on SIIM-ACR, an independent dataset with radiologist-drawn masks, where the failing models detect 0 of 432 confirmed pneumothoraces, including 0 of the 144 largest lesions. Reading the models' logits and internal states shows this is not blindness. The two failing models are highly confident yet no better than chance and a linear probe recovers pneumothorax from their hidden representations even where recall is zero, so the finding is represented internally but suppressed at the output. Aggregate accuracy therefore cannot establish that a medical VLM can see a given finding, because a model that misses one can still describe it convincingly. Reporting per-finding recall and specificity on balanced cases is needed to tell the difference.
本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：MedGemma、HuatuoGPT、Qwen2.5-VL、ChestX-ray14，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.LG」（@Shreya Ajay Kale, Angad Ripudaman Singh Bajwa），经 AITOP 多维评分管线结构化提炼。
