论文定位 VLM 跨图像对比失败源:16-32 个注意力头,微调 0.02% 参数可修复
Seen but Unsaid: Locating and Repairing Cross-Image Comparison in Vision-Language Models
Qwen3-VL、InternVL3.5 这类 VLM 跨图对比老出错,这篇论文把锅找到了具体 16-32 个注意力头上,只训 0.02% 参数就能把 SPair-71k 提升 28 个点。做视觉模型微调或可解释性研究的人值得跑一下他们的验证脚本。
arXiv 论文《Seen but Unsaid》系统追踪了视觉语言模型(VLM)在跨图像对比任务上的失败根源,覆盖 Qwen3-VL(2B-32B)、InternVL3.5-8B 和 Gemma-3-12B 三个模型家族,任务包括合成形状匹配、HPatches 点对应和 SPair-71k 语义对应。研究发现两个独立失败点:一是答案位置探针的准确率比校准后的模型输出高出 15 到 25 个点,说明模型内部已经编码了正确答案但未能读出;二是语义对应信息存在于视觉 token 中却从未传递到答案层。
机制上,作者定位到一小组 16-32 个注意力头承担跨图像比较功能: redistributing 这些头的跨图像注意力(保持质量守恒)会使合成任务准确率下降 17-22 个点、HPatches 下降 15-32 个点,而随机或注意力匹配的对照头最多只造成 7 个点下降。仅训练这些头的 query 和 key 投影——约占 0.02% 参数——就把 SPair-71k 准确率提升 28.3 个点(Qwen3-VL-8B)和 25.1 个点(InternVL3.5-8B),并迁移到 HPatches 和 BLINK 基准。
对开发者的意义在于:全层微调 adapter 有 21-51% 的增益是通过同一组注意力头的注意力变化实现的,且主要改变这些头读取的内容而非权重本身。这意味着针对特定头的定向微调可能比全参数训练更高效。论文所有决策规则在实验前冻结,所有数字由独立验证脚本复算。
Seen but Unsaid: Locating and Repairing Cross-Image Comparison in Vision-Language Models
Vision-language models (VLMs) often fail to match a point or an object across two images, even when the answer can be decoded from their own hidden states. We trace this failure from the visual tokens to the answer letter, in Qwen3-VL (2B-32B), InternVL3.5-8B, and Gemma-3-12B, on synthetic shape matching, same-instance point correspondence (HPatches), and semantic correspondence (SPair-71k), and find two distinct failure points, a small set of attention heads that carries the comparison, and a repair that touches 0.02% of the parameters. Two failure points: exact copies are compared but not read out. An answer-position probe exceeds the calibrated output by 15 to 25 points in every family, for nameable and unnameable shapes alike, and steering along the probe's direction moves no more answers than random directions. Semantic correspondences are present in the visual tokens but never reach the answer. One pathway: redistributing the cross-image attention of 16-32 heads, with its mass preserved, removes 17-22 points of synthetic and 15-32 points of HPatches accuracy, against at most 7 for random or attention-matched heads. A lever that training finds on its own: training only these heads' query and key projections raises SPair-71k accuracy by 28.3 points (Qwen3-VL-8B) and 25.1 points (InternVL3.5-8B), beats both layer-matched and attention-matched heads, and transfers to HPatches and BLINK. An unconstrained all-layer adapter routes 21-51% of its gain through the attention of the same heads, against at most 7% for random heads, and does so by changing what they read (83-86% of the effect) rather than their weights. Every decision rule was frozen before its run, and every reported number is recomputed by independent verification scripts.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Qwen3-VL、InternVL3.5、VLM、可解释性,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Zhaoning Zhang),经 AITOP 多维评分管线结构化提炼。