论文精选

研究揭示 VLM 描述图像时如何规划物体列举顺序

How VLMs Plan What's Next

精选理由

这篇论文把 Qwen3-VL 生成描述时的内部机制拆开了:怎么定位物体、怎么记住写到哪里、怎么决定下一个看哪。用因果补丁验证,2B 到 32B 都成立。做可解释性或多步规划研究的读者值得拿来当起点。

一篇 arXiv 论文通过因果补丁实验刻画了 Qwen3-VL 模型在描述图像时的规划机制,回答了视觉语言模型如何决定"下一个看哪里"的问题。该机制包含四个环节:binding heads 将相关文本 token 与图像匹配并解析为指向图像区域的指针;同样的 binding heads 把已写出的答案绑定到来源区域,形成类似光标的进度信号,标记列表已写到图像的哪个位置。

正文

接下来,select 步骤在光标之后选择最近的未写区域或决定停止,gaze heads 则执行选定的指针并返回该区域的内容概念,再由语言 heads 和后期 MLP 转化为回答中的下一个词。验证实验在 Qwen3-VL 上基于合成视觉检索任务进行,并通过与替代假设对比来排除其他解释。

结果显示该机制在多种输入格式、自然照片和不同语言下均成立,binding、relay 和 gaze heads 在 2B 到 32B 的所有模型规模上都存在。项目页面位于 plan.baulab.info,这项工作为逐层、逐 token 地追踪 VLM 如何把问题和图像转化为答案序列提供了方法。

原文 · arXiv cs.LG

How VLMs Plan What's Next

When a vision-language model describes an image, it names one object at a time and it rarely repeats itself, yet how the model plans this order is far from obvious. We characterize the mechanism a VLM uses to decide where to look before it looks. The mechanism has four main parts. First, binding heads match the relevant text tokens against the image and resolve them into pointers to the matched image region. Second, the same binding heads bind each written answer to the region it came from, and the model turns it into a progress signal: a cursor, the position in the image up to which the list has been written. Third, a select step chooses the nearest unwritten region after the cursor, or stops. Finally, gaze heads execute the chosen pointer and return a concept of what is drawn there, which language heads and the late MLPs turn into a word in the language of the question. We validate each part of this mechanism through causal patching in Qwen3-VL models on a synthetic visual retrieval task, and test our mechanisms against alternative hypotheses. We further show that the mechanism holds across input formats, natural photographs, and languages, and that its binding, relay, and gaze heads appear at every model size from 2B to 32B. This work allows us to investigate how a VLM turns a question and an image into a sequence of answers, layer by layer and token by token. More broadly, it opens a way to study how models plan over complex multi-step tasks with long rollouts. Our project is available at https://plan.baulab.info/

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Qwen3-VL、VLM、可解释性、视觉语言模型,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Rohit Gandikota, Arnab Sen Sharma, Adam Belfki, David Bau),经 AITOP 多维评分管线结构化提炼。