论文精选

Transformer模型通过位置追踪和目标指南实现世界建模

World Modeling in Transformers

精选理由

这个研究很酷,它用TaxiGPT模型展示了Transformer如何通过位置追踪和目标指南来模拟世界,还解释了模型失败的原因。

研究通过机械分析证明,在曼哈顿随机漫步训练的TaxiGPT模型能够追踪自身位置并使用目标指南进行导航。模型失败源于交叉路口特征干扰,而‘功能打包’机制有助于缓解此问题。

正文

研究提出机械指标来比较模型,并发现世界建模能力在训练不同阶段出现。

原文 · arXiv cs.AI

World Modeling in Transformers

Behavioral failures can make a transformer appear to lack a world model even when it has learned faithful representations of its environment. We demonstrate this in TaxiGPT, a transformer trained on random walks through Manhattan whose failures have been interpreted as evidence of an incoherent internal map. Through mechanistic analysis and causal interventions, we show that the model represents intersections and streets, tracks its position, and uses a goal compass to navigate. We trace its failures to interference between superposed intersection features, which disrupts localization within the internal map. Affordance packing, which groups representations of intersections with the same legal moves, helps limit the consequences of these errors. Finally, we propose mechanistic indicators that we use to compare models and show that world-modeling capacities emerge at different stages of training. Our findings motivate a shift from asking whether a model has a world model to mechanistically studying its world modeling: the interacting capacities through which it represents its environment and uses those representations to guide behavior.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「各国 AI 监管法案、合规治理与主权算力政策」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Transformer、TaxiGPT、世界建模、机械分析,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Pierre Beckmann, Matthieu Queloz, Andre Freitas),经 AITOP 多维评分管线结构化提炼。