研究揭示 MLLM 视觉-文本对齐分数可能是几何假象
The Alignment Illusion in Multimodal Large Language Models
有人测试了 13 个多模态模型，发现 CKA 这些常用的对齐指标根本测不出视觉信息是否真被用上，还给出了新指标 PA gap，做可解释性分析的值得看看。
一篇 arXiv 论文对 13 个 MLLM（来自 5 个模型家族、参数规模 0.5B 到 72B）做受控干预：把投影器输出的视觉 token 换成高斯噪声后任务准确率大幅下降，但 CKA、SVCCA、MIR 和主角度余弦四个标量指标都无法稳定区分受损流和原始流。作者把这种失效命名为 alignment illusion，归因于语言模型通路中各向异性的 MLP 下投影把视觉和文本 token 拉向共同输出方向，形成权重诱导的对齐。论文据此提出 principal-angle gap（PA gap），即前两个主角度余弦之差，实验显示它在分级视觉损坏下比标量分数更能追踪任务准确率。
The Alignment Illusion in Multimodal Large Language Models
Layer-wise visual-text similarity in Multimodal Large Language Models (MLLMs) is widely interpreted as evidence that the language model progressively integrates visual content into a shared representation space. This reading rests on the assumption that scalar alignment scores reflect content-level cross-modal interaction. To test this assumption, we apply controlled interventions to the visual stream. Across 13 MLLMs from five families spanning 0.5B to 72B parameters, replacing projector-output visual tokens with Gaussian noise sharply reduces task accuracy, yet four standard scalar measures (CKA, SVCCA, MIR, and the leading principal-angle cosine) fail to consistently separate the corrupted stream from the original. We call this failure the alignment illusion and trace it to the shared language-model pathway: anisotropic MLP down-projections pull visual and text tokens toward common output directions, producing weight-induced alignment. Because this component is essentially one-dimensional, we introduce the principal-angle gap (PA gap), defined as the difference between the top two principal-angle cosines, which separates weight-induced similarity from multi-directional visual structure. Under graded visual corruption, the PA gap tracks task accuracy more consistently than the scalar scores we consider; under a structured but irrelevant image, it further exposes regimes in which internal geometry and task accuracy come apart. Internal visual-text alignment in MLLMs is therefore best read as a geometric diagnostic of the visual stream inside the language model rather than a direct proxy for content-level cross-modal interaction, and is most informative when calibrated by controlled task evidence.