论文

V-lens:透过统一多模态模型的隐藏状态可视化推理概念

The Elephant in the Question: Unified Models Picture the Concepts Behind Their Answers

精选理由

给做多模态模型可解释性研究的人:这篇把隐藏状态直接画成图来看模型在想什么,不用训练解码器。lens tuning 把代词指代可视化率从 2-11% 拉到 86-94%,还顺带提升了文本答题,思路挺新。

arXiv 论文《The Elephant in the Question》提出 V-lens,无需训练解码器即可将统一多模态模型中文本 token 的隐藏状态可视化成图像,覆盖 4 个扩散架构与 2 个纯自回归的统一模型。其机制是利用统一架构中图像生成器可直接访问文本处理阶段表征这一特点,把中间概念画出来,即使这些概念在回答中从未被显式提及。

正文

论文进一步提出 lens tuning,用图片对选定的隐藏状态做监督。在 BAGEL、SenseNova-U1 和 HunyuanImage-3.0 上,代词指代对象的正确可视化率从 2-11% 提升到 86-94%,且这种视觉监督还能间接提高文本回答质量。受控实验展示了可视化内容随上下文、token 位置和深度的变化规律。

对研究者而言,这提供了一条检查模型推理中间步骤的可解释性路径,代码已在 GitHub 开源。

原文 · arXiv cs.LG

The Elephant in the Question: Unified Models Picture the Concepts Behind Their Answers

Visualizing intermediate representations could help us inspect the reasoning steps underlying a model output. Unified multimodal models offer this opportunity by combining understanding and generation in one architecture. Their image generators can access representations formed while processing text. We introduce the V-lens, which uses this connection to visualize a chosen token's hidden state without training a decoder. Across four diffusion-based and two purely autoregressive unified models, the resulting images reveal key intermediate concepts that connect a question to its answer even when the concepts are never named explicitly. Controlled tests show how this content changes with context, token position, and depth. We then introduce lens tuning, which supervises selected hidden states with pictures through the V-lens. Across BAGEL, SenseNova-U1, and HunyuanImage-3.0, the rate of correct visualizations of pronoun referents rises from 2-11% to 86-94%. Surprisingly, such visual supervision also improves text answers without supervising the answer tokens directly. Code is available at https://github.com/multimodal-ai-lab/V-lens.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦中国 AI 产业与国产大模型生态。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:V-lens、BAGEL、HunyuanImage-3.0、多模态,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Tobias Braun, Marcus Rohrbach, Lorenzo Cavallaro),经 AITOP 多维评分管线结构化提炼。