模型

AURA 发布:音频驱动的流式数字人视频模型,支持实时低延迟生成

AURA: AUdio-dRiven streaming Avatar

精选理由

语音助手和虚拟主播要低延迟,现有双向扩散模型要等好几秒音频才能出画面。AURA 用滚动 KV cache 做到当前块音频一进来就开始生成,1.3B 单卡就能跑 576p 实时,5B 能到 720p。权重、30K 视频数据集全开源,做交互式数字人的开发者可以直接上手试。

论文提出 AURA,一套音频驱动的流式数字人视频生成模型,提供 1.3B 和 5B 两个规格。AURA 1.3B 可在单张 GPU 上以 20.7 fps 实时流式生成 576p 视频(对应 16 Hz 播放帧率),AURA 5B 则以 9.7 fps 生成 720p 视频。

正文

技术上,AURA 采用分块生成加滚动 KV cache,只需当前块的音频即可开始生成,无需双向模型所需的多秒音频前瞻。训练上采用两阶段蒸馏:先用块状滑窗因果注意力掩码和更少扩散步数对齐因果学生模型与教师轨迹,再用带 backward simulation 的 Distribution Matching Distillation(DMD)缩小质量差距。

基准成绩方面,AURA 1.3B 比其双向教师模型快 23 倍,质量与 TalkVerse 在较低分辨率下相当;AURA 5B 在 720p 下与 TalkVerse 教师持平且快 17 倍。团队已开源两个模型的权重、30K 条合成视频数据集及预计算的教师轨迹。

原文 · arXiv cs.AI

AURA: AUdio-dRiven streaming Avatar

Current audio-driven video diffusion models can synthesize realistic talking-head videos but generally require dozens of diffusion denoising steps over multi-second video chunks, making them unsuitable for streaming applications such as voice assistants and virtual avatars, where low-latency and online causal generation are required properties. AURA generates video chunk by chunk with a rolling KV cache, requiring only the audio of the current chunk to start generating, rather than a multi-second audio look-ahead like bidirectional models. It comes in two sizes: AURA 1.3B streams 576p video in real time on a single GPU (20.7 fps for a 16 Hz playback rate), making it suited to interactive avatar applications, and AURA 5B streams 720p video at 9.7 fps. Our two-stage distillation pipeline first aligns the causal student with the teacher, by training it to match the teacher's trajectories with fewer diffusion steps and a chunk-wise sliding window causal attention mask, then applies Distribution Matching Distillation (DMD) with backward simulation to close the remaining quality gap. On standard talking-head benchmarks, AURA 1.3B runs 23x faster than its own bidirectional teacher and reaches a quality comparable to TalkVerse at a lower resolution, and AURA 5B matches its TalkVerse teacher at 720p while running 17x faster. To support reproducibility and further research on streaming avatar generation, we release the weights of both models along with a dataset of 30K synthetic videos and their precomputed teacher trajectories.

产业坐标与信源核验

本条情报归属 AITOP「模型」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:AURA、数字人、视频生成、流式生成,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Nathaniel Cohen, Nicolas Dufour, Amélie Royer, Alasdair Newson, Patrick Pérez),经 AITOP 多维评分管线结构化提炼。