论文

Cycle WAM:用循环一致性耦合动作与视频建模

Cycle WAM: Coupling Action and Video through Cycle Consistency

精选理由

机器人领域的新框架 Cycle WAM,让视频和动作双向对齐,LIBERO-Pro 上拿了同类最好成绩,还能改进 DreamZero。

Cycle WAM 把视频和机器人动作视为由因果关系连接的成对模态,在统一训练框架中学习二者对应关系。该方法先通过双向转换增强跨模态对应,恢复正向与逆向动力学,再引入可微的循环重建目标进一步耦合两个映射。模型在 LIBERO-Pro 上达到同类方法的最优表现,并在 DROID 上显著改进 DreamZero。与 UWM 的对比显示其逆向动力学表现更优,正向动力学预测的画面细节更清晰。

原文 · arXiv cs.AI

Cycle WAM: Coupling Action and Video through Cycle Consistency

Recent advances in video modeling have driven substantial progress in world action models (WAMs). Existing approaches exploit pretrained video priors by decoding actions from latent representations or inferring them from generated videos. The underlying link is causal: robot actions produce changes in the scene, and video observations of these changes provide evidence for inferring the actions that produced them. However, existing training formulations do not by themselves ensure that actions and their predicted visual consequences are consistent, potentially limiting the use of pretrained capabilities. We treat video and action as paired modalities linked by this causal relationship and learn their correspondence through a unified training framework designed to better exploit pretrained video priors. Cycle WAM first augments video-action co-training with bidirectional conversion to reinforce cross-modal correspondence and recover forward and inverse dynamics. It then introduces a differentiable cycle-reconstruction objective to further couple the two mappings. The resulting model supports both forward and inverse dynamics, with the potential to improve performance through more effective use of pretrained video priors. Cycle WAM achieves state-of-the-art performance among comparable methods on LIBERO-Pro and substantially improves DreamZero on DROID. Comparisons with UWM characterize its inverse-dynamics performance and show sharper visual detail in the displayed forward-dynamics predictions.