Roz 评测揭示世界模型潜变量缺失物理属性
Roz: A Taste Test for World Model Latents
这篇论文给世界模型出了道物理题：看着像懂物理，其实连摩擦力都没记住。Roz 评测 1734 个场景一次跑完，做世界模型的同学可以拿来自测。
arXiv 论文提出 Roz 评测，检验主流 latent dynamics 世界模型是否真正编码物理规律。评测包含 10 个对比任务和 3 个连续任务，共 1734 个场景，结果显示现有模型在图像相似度类任务上表现强，但丢失物体质量、形变、摩擦力和弹性等细节。Roz 只需模型能输入静帧或视频、提取潜变量并提供距离函数即可一次运行完成。作者还在自研模型 AnonWM 上验证，发现性能随训练步数提升，但与重建 MSE 不一定相关。
Roz: A Taste Test for World Model Latents
State of the art latent dynamics world models achieve impressive performance for robot planning. But do they actually encode the physics of the world? In this paper, we show that popular world models today achieve strong results on tasks that reduce to image similarity but forget key details about objects in the scene like mass, deformation, friction, and bounciness. We achieve this by defining precisely what ought to be encoded in such a world model and introducing two new evaluation methods that can directly tease physical properties and easily scale to new ones. Our evaluation, which we call Roz, includes 10 contrastive tasks and 3 continuous tasks, totaling 1734 individual scenarios. We validate our eval on control tasks. On our own model, AnonWM, we find that performance roughly improves with training steps, while not necessarily being correlated with reconstruction MSE. We argue current model failures are a symptom of evaluation strategies (probes and planners) that are opaque, model-specific, and high-compute. Roz requires only a model that can ingest still frames or video, a way to extract latents, and a distance function between them, and it can be run in one pass. We hope future evaluations will adopt Roz's evaluation techniques. More broadly, we hope Roz can be a case study for a future where benchmarks allow rapid iteration and comparison across models.
