LynnReal-Hand:用手部动作直接操控的交互式视频世界模型
LynnReal-Hand: Unlocking World Interaction with Hand Control
用手就能直接操控视频世界模型,不再只靠文字和镜头指令。单张 H200 跑到 18 FPS、延迟 741ms,还能实时交互。做具身智能、交互仿真或者世界模型研究的开发者值得关注,代码和数据都会开源。
一篇 arXiv 论文提出 LynnReal-Hand,一个通过手部控制直接与视频世界模型交互的系统。团队整理了 100K 条覆盖真实与合成环境的手部交互片段,并设计手部感知监督来提升细粒度动作与物体交互的建模质量。
技术上有三个关键组件:用因果流映射实现少步生成,用区域感知分数蒸馏融合视频与图像先验来优化交互动态,并推出轻量视频解码器 Ultra-Fast-VAE 降低推理开销。在 HandBench 上,完整序列模型取得 FVD 181.67、MANO Dice 93.02,三步因果模型的 MANO Dice 也达到 92.49。
实时性能方面,单张 H200 上 540p 流式生成达到 18.03 FPS,延迟 741ms,峰值显存仅 34.89GB。团队承诺开源数据集、模型权重与完整训练推理代码,可支撑手控世界模型方向的后续研究。
LynnReal-Hand: Unlocking World Interaction with Hand Control
Interactive video world models enable exploration through text and camera controls, yet direct object interaction requires finer control. Hands provide a natural interface, but translating continuous hand movements into coherent visual responses remains challenging because of complex hand-object dynamics and demanding real-time computation. To address these challenges, we introduce LynnReal-Hand, a hand-controlled video world model that unifies precise interaction modeling with efficient streaming generation. We curate 100K interaction clips spanning diverse real and synthetic environments and develop hand-aware supervision to improve fine-grained hand motions and object interactions. Building on this foundation, we learn a causal flow map for few-step generation and introduce region-aware score distillation that combines complementary video and image priors to refine interaction dynamics and visual appearance. To further reduce inference latency, we propose Ultra-Fast-VAE, a compact video decoder that preserves high reconstruction fidelity with substantially lower computational cost. Extensive evaluations on HandBench demonstrate superior hand-following accuracy and visual quality compared with existing approaches. Our full-sequence model achieves an FVD of 181.67 and a MANO Dice score of 93.02, while the three-step causal model maintains a MANO Dice score of 92.49. For real-time interaction, 540p streaming on a single H200 reaches 18.03 FPS with 741ms latency and an optimized peak GPU memory footprint of only 34.89GB. We will release the datasets, model weights, and the full training and inference code to support research on hand-controlled video world models.
本条情报归属 AITOP「论文」核心栏目,重点聚焦中国 AI 产业与国产大模型生态。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:LynnReal-Hand、世界模型、视频生成、HandBench,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Shaohao Rui, Chaoyang Zhang, Xiaofeng Mao, Weijie Ma),经 AITOP 多维评分管线结构化提炼。