CueClash 测试揭示前沿 VLM 依赖 2-D 线索判断 3-D 关系
CueClash: Frontier Vision-Language Models Judge 3-D Relations Close to the Picture Plane
想搞懂 VLM 到底能不能真正理解空间,这篇设计得很巧:故意让 2-D 线索一半是坑,结果 Opus 5.5、GPT-5.5、Gemini 3.6 Flash 全部中招,掉 18–37 个百分点。做空间理解或具身的开发者值得看它怎么用深度信息补救。
arXiv 论文 CueClash 用渲染场景区分“从图片判断”与“从地面布局判断”两种 3-D 关系推理,五类关系中被试陈述里 2-D 线索恰好一半正确、一半误导。Claude Opus 5.5、GPT-5.5 和 Gemini 3.6 Flash 在低推理档下的 shortcut gap 达 +18 到 +37 个百分点,剔除两类混淆项后仍有 +7 到 +29。给定地面坐标和关系定义时,三款模型答题正确率达 99%–100%,说明失败发生在从图片到布局的推断环节。
机制分析显示,模型俯视重建的场景图大多保留深度顺序,但和图片一样压缩了深度。GPT-5.5 的地图几乎照搬图片,Opus 5.5 和 Gemini 的地图修正约 0.6 的深度压缩,Gemini 又额外均匀缩小所有深度。用像素测量加相机深度能解决大部分“物体能否穿过开口”类问题。三款模型在 CV-Bench 照片上遇到误导性 2-D 线索时准确率同样下降。
对开发者的含义是:VLM 的空间推理仍以图像表观线索为主,给模型补充显式几何信息(深度、坐标)能显著改善结果。两款开源模型的布局信息虽传到了语言模型,但地图重建阶段丢失大量信息。
CueClash: Frontier Vision-Language Models Judge 3-D Relations Close to the Picture Plane
A vision-language model (VLM) can judge 3-D relations from the picture or from the ground layout. In photographs the two usually agree. We separate them with CueClash, rendered scenes in which the 2-D cue for each of five relations is right on exactly half the statements. The shortcut gap is accuracy when the cue is right minus accuracy when it is wrong. For Claude Opus 5.5, GPT-5.5 and Gemini 3.6 Flash at low reasoning effort it is +18 to +37 percentage points. It is +7 to +29 without two relations whose cue-wrong items are confounded with borderline cases or occlusion. Given ground coordinates and each relation's definition, they answer 99 to 100% correctly. Pixel measurements with camera depths remove most of the gap for objects fitting through openings. The failure lies in the step from the picture to the layout. The maps that the models draw of the scene from above mostly keep the depth order but, like the picture, shorten depth. GPT-5.5's maps stay close to a copy of the picture. Opus 5.5's and Gemini's maps undo about 0.6 of this shortening. Gemini's then shrink all depths by a further uniform factor. Opus 5.5 follows the picture mainly on fits, and its maps correct depth about as much as human observers of real scenes do. The three models also lose accuracy on CV-Bench photographs where a reliable 2-D cue misleads. In two open models the layout information reaches the language model, but their maps lose much of it.
本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Claude Opus 5.5、GPT-5.5、Gemini 3.6 Flash、VLM,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Soham Dan),经 AITOP 多维评分管线结构化提炼。