EgoHandWAM 论文:手部动作在第一人称世界动作建模中的作用
EgoHandWAM: Probing the Role of Hand Motion in Egocentric World Action Modeling
训练了75万个EgoDex片段,量化手部轨迹到底能帮第一人称模型预测多少未来,答案是不多:误差只降几个百分点。做具身智能或视觉预测研究的同学值得看数字,省得自己再跑一遍消融。
EgoHandWAM 是一篇新论文,研究第一人称视频中手部运动对未来视觉内容的预测价值。模型从当前图像和近期手部状态预测三维手腕与指尖轨迹及未来视觉特征,训练数据为 751,065 个 EgoDex 窗口,视觉编码使用冻结的 DINOv2。在保留的测试会话上,纯动作模型将两秒后手腕端点误差从静止预测基线的 60.26 mm 降到 56.48 mm,但对长距离伸手动作仍存在低估。
实验结果显示手部信息的增量价值有限:加入未来特征监督后轨迹精度几乎不变;视觉预测头借助真实手部历史仅将 smooth L1 误差降低 1.32%,且将历史跨会话打乱后增益完全消失。在另一组四任务探针实验中,加入真实未来位移和未来追踪有效性掩码可将视觉特征误差降低 3.93%,但只用掩码几乎无效,用模型自己预测的轨迹也几乎没有帮助。
迁移实验同样偏冷:在两个 RoboTwin 仿真任务上,用对齐的人手示例替换一批机器人示例,跨三个训练种子没有带来一致的任务成功率提升。整体结论是手部运动对第一人称世界建模有可测但幅度不大的贡献。
EgoHandWAM: Probing the Role of Hand Motion in Egocentric World Action Modeling
How much does hand motion tell us about what a first-person camera will see next? EgoHandWAM forecasts three-dimensional wrist and fingertip trajectories and a future visual feature from the current image and recent hand states. We train on 751,065 EgoDex windows using a frozen DINOv2 encoder. On held-out development sessions, the action-only model reduces two-second wrist endpoint error from 60.26 to 56.48~mm relative to a stationary forecast. It still underestimates long reaches. Adding future-feature supervision gives nearly identical trajectory accuracy. With the recorded hand history, the visual head reduces smooth L1 error by 1.32\% relative to copying the current feature. Shuffling histories across sessions erases this gain. In a separate four-task probe, adding recorded future displacements to observed context and future tracking-validity masks reduces visual-feature error by 3.93\%. Masks alone give little improvement. A trajectory predicted from the same observations adds little. On two simulated RoboTwin tasks, substituting aligned human examples for an auxiliary batch of robot examples gives no consistent increase in task success across three training seeds.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:EgoHandWAM、EgoDex、DINOv2、具身智能,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Kailin Wang, Haoxiang Jie, Yaoyuan Yan, Zhiyou Heng, Zhaosong Li),经 AITOP 多维评分管线结构化提炼。