论文

单张参考图即可定制人物视频生成:测试时自监督个性化方法

One Image Is Enough: Self-Generated Supervision for Personalizing Human Video Diffusion at Test Time

精选理由

一张照片就能让人物视频模型保住脸和衣服,不用录视频做微调,代码和权重都开源了,做视频生成的可以试试。

该方法针对姿态驱动的人物视频扩散模型在姿态偏离参考图时身份丢失的问题,用一张参考图完成个性化。流程分三步:域调优的姿态条件生成器在关键姿态合成伪真值帧,推理时的 reference-only 采样器筛选身份最一致的候选帧,再为每个对象训练低秩适配器(LoRA)。全程不需要对象的多姿态视频、真值或标注。在 TikTok 和 UBC Fashion 基准上,适配后模型在所有指标上超过基线,仅用 0 张真实视频帧即可媲美或超过用 5 帧真实数据做适配的效果。代码与权重已在 GitHub 开源。

原文 · arXiv cs.LG

One Image Is Enough: Self-Generated Supervision for Personalizing Human Video Diffusion at Test Time

Pose-driven human video diffusion animates a person from a single reference image and a target pose sequence, yet it struggles to preserve identity, rendering the face and garments unfaithfully as the pose departs from it. Personalization is the standard remedy, but its supervision is missing: one image fixes identity at a single pose, while a multi-pose subject video cannot be assumed in practice. Instead, we synthesize the missing supervision. A domain-tuned, pose-conditioned generator produces pseudo-ground-truth frames at selected key poses. An inference-time reference-only sampler steers these candidates toward the subject and keeps the most identity-consistent. A per-subject low-rank adapter then trains on static clips of them, configured from the reference alone, with no subject video, ground truth, or labels. Although lower in fidelity, these synthetic frames supervise more effectively than real ones. Real frames as captured carry motion that erodes the temporal prior, while a static clip conveys appearance alone. On TikTok and UBC Fashion, the adapted model improves the base on every metric, inverts the identity-versus-motion trade-off, and matches or surpasses adaptation on five real frames despite using none. Code and weights are available at https://github.com/SadmanPinon/one-image-is-enough.