WorldCrafter:带可查询3D记忆的视频世界模型
WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory
给视频生成器配了个能按视角查的3D记忆库,单张图逛场景一分钟不穿帮,做世界模拟的可以看看这篇。
WorldCrafter 是一个视频世界模型,核心是为生成器配备可按相机视角查询的隐式 3D 感知记忆。记忆编码器和位姿条件读取模块在去噪前,把历史多视角观测压缩成一组固定的目标视角专用 token,训练时与视频生成器联合优化,不依赖显式深度对应关系。结合近期时序上下文和少步蒸馏后,WorldCrafter 能从单张输入图片或一段文字提示出发做流式场景探索。在静态和动态场景实验中,它在分钟级探索里明显提升了长程一致性和相机控制精度,同时保住了画面质量。
WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory
Video world models enable interactive exploration of dynamic environments, yet struggle to respect prior observations over long horizons and across viewpoints. We present WorldCrafter, a video world model that learns a camera-queryable implicit 3D-aware memory for this purpose. The key insight is to let the requested viewpoint shape how multi-view evidence is compressed into the video generator's limited token budget. Trained jointly with the video generator, a memory encoder and pose-conditioned readout module integrate historical observations into a fixed set of target view-specific tokens before denoising, without explicit depth-based correspondences. By combining this memory with recent temporal context and few-step distillation, WorldCrafter enables streaming scene exploration from a single input image or text prompt. Experiments across static and dynamic scenes show substantial gains in long-horizon consistency and camera-control accuracy while preserving visual quality during minute-scale exploration.