DoF-Query Transformer 融合视觉与前臂 HD-sEMG,连续估计手部 20 关节角度
DoF-Query Transformer for Adaptive Fusion of Egocentric Vision and Forearm HD-sEMG in Continuous Hand Kinematics
手部动捕不用戴数据手套了:头戴相机加前臂肌电贴片就能估出 20 个关节角度,R² 到 0.88,遮挡视角下也扛得住。做机器人示教数据采集或人机交互的团队可以看看这个融合思路。
一篇发表于 arXiv 的研究提出多模态框架,从自我中心 RGB 图像和 128 通道前臂高密度表面肌电(HD-sEMG)连续回归手部 20 个关节角度,无需在手指上安装传感器。框架由 Vision Transformer 视觉编码器、Transformer 肌电编码器和 DoF-Query Transformer 融合解码器组成:解码器为每个自由度分配独立可学习 query,实现按 DoF、按输入条件自适应地融合两种模态。
在 10 名参与者的同步多模态数据上评估,融合模型平均 R² 达 0.88、MAE 为 2.80°,优于仅视觉(0.82 / 3.21°)和仅肌电(0.68 / 5.13°)模型,也超过受测的单模态基线和其他融合设计。按 DoF 分析显示,视觉模糊的关节和视角下融合增益更大;合成视觉遮挡实验也证明其比纯视觉估计更稳健。
这项工作的应用方向是采集细粒度人类操作示范,为具身智能数据集和灵巧机器人学习提供动作监督信号,也可用于直觉化人机交互。
DoF-Query Transformer for Adaptive Fusion of Egocentric Vision and Forearm HD-sEMG in Continuous Hand Kinematics
Continuous and accurate estimation of human hand kinematics is important not only for intuitive human-machine interaction, but also for capturing fine-grained human manipulation demonstrations that can provide action supervision for embodied-intelligence datasets and dexterous robot learning. However, individual sensing modalities may be insufficient for reliable estimation under varying operating conditions. We present a finger-unencumbered multimodal framework for continuous regression of 20 hand joint angles from egocentric RGB images and 128-channel forearm high-density surface electromyography (HD-sEMG). The framework combines a Vision Transformer-based visual encoder, a Transformer-based EMG encoder, and a degree of freedom (DoF) query Transformer fusion decoder. To account for the varying informativeness of visual and EMG cues across joint DoFs and input conditions, the decoder assigns a separate learnable query to each DoF, enabling DoF-specific and context-adaptive fusion of the two modalities. The framework was evaluated on synchronized multimodal data from 10 participants. Averaged across participants, the proposed fusion model achieved a mean coefficient of determination ($R^2$) of $0.88$ and a mean absolute error (MAE) of $2.80^\circ$, compared with $0.82$ and $3.21^\circ$ for the vision-only model and $0.68$ and $5.13^\circ$ for the EMG-only model. It also outperformed the evaluated representative unimodal baselines and alternative fusion designs. DoF-wise analyses showed larger fusion gains for visually ambiguous joints and viewing conditions, while synthetic visual occlusion demonstrated greater robustness than vision-only estimation. These results highlight the complementary value of egocentric vision and forearm HD-sEMG for capturing fine-grained human hand kinematics, with potential applications in human demonstration acquisition and dexterous robot learning.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:HD-sEMG、Vision Transformer、手部动捕、具身智能,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Jiahang Liu, Wenbo Zhang, Junsong Wang, Zhenzhi Ying, Siqi Hou, Naohiko Sugita, Liming Shu),经 AITOP 多维评分管线结构化提炼。