DominoBench 发布:30 个多模态模型看图预测物理连锁反应无一过关
DominoBench: Evaluating Multi-Step Physical Reasoning from Still Images
看图预测多米诺连锁反应,30 个模型全军覆没。GPT、Gemini、Claude 这些旗舰在纯看图时最多 72% 命中,但把决定性计算写出来后能到 82-92%,说明瓶颈在定量推理。做具身智能或机器人规划的开发者值得用这个基准测测自己的模型。
DominoBench 是一个基于物理模拟器判定结果的新基准,包含 854 道题、427 组最小对,测试多模态模型能否从静态图片预测斜坡、摆锤、跷跷板和多米诺骨牌等 2D 刚体场景的连锁反应。团队测试了 10 个模型家族共 30 个模型,结论是没有任何模型能在无辅助条件下可靠预测结果。
机制上,每道题设计为改变 1-3 个格子的某个量就决定一个人是否幸存,并配有四级提示阶梯,最后一级给出决定结局的精确计算。Anthropic、Google 和 OpenAI 的前沿模型在无提示时对关键人物的判断正确率不超过 72%,成对题目同时答对的比例不超过 34%。给出文本形式的精确几何信息帮助甚微,但当把决定性计算及其所需量(包括静态图无法显示的速度)写出后,同一批模型达到 82-92%,而小型开源模型仍接近随机水平。
这说明失败出在定量计算环节:最强模型能在被给出决定性量时使用它,却无法自行得出。对需要规划或作用于物理世界的模型而言,流畅的场景描述并不能证明预测正确,DominoBench 正是检验模型能否自主找到并计算决定性量。
DominoBench: Evaluating Multi-Step Physical Reasoning from Still Images
Can a vision-language model look at a picture of a chain reaction and tell what will happen? Recent multimodal models describe scenes fluently and solve textbook physics problems, which suggests they should also predict how a scene of ramps, ledges, pendulums, seesaws and dominoes plays out once a ball is pushed. We tested this claim on 30 models from 10 families and reached a clear conclusion: unaided, no model predicts the outcome reliably. We built DominoBench, a benchmark of still images of 2D rigid-body scenes whose outcome a physics simulator computes. It holds 854 items in 427 minimal pairs, in which a change of one to three grid steps in one quantity decides whether one person survives, and every item comes with a four-rung hint ladder ending in the exact calculation that decides the outcome. Unaided, frontier models from Anthropic, Google and OpenAI are right about the person whose fate flips in at most 72% of items and answer both twins of a pair correctly in at most 34%. Pointing at the deciding step helps little, and exact geometry given as text barely helps, yet once the deciding calculation and the quantities it needs, including speeds that a still image cannot show, are written out, the same models reach 82-92% on the deciding person, while small open-weight models stay near chance. Physical prediction from a still image therefore fails at the quantitative step: the strongest models use the deciding quantity when they are given it, but do not produce it on their own. For models that plan or act in the physical world, a fluent account of a scene is thus no evidence of a correct prediction: such models must find and compute the deciding quantity unprompted, and DominoBench tests whether they do.
本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:DominoBench、多模态、物理推理、基准测试,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始信息发布自一手权威信源「arXiv: OpenAI」(Tier-1 一手发布),具备高确定性参考价值。