论文

GS-VTON-MVT:用多视角 Transformer 与几何引导 ControlNet 实现 3D 虚拟试穿

Multi-View Transformer with Geometry-Guided Multi-Modal ControlNet for 3D Virtual Try-On

精选理由

3D 扫描的人物换衣服一直是老大难,各个视角改出来对不上。这篇论文把跨视角一致性做成模型自己学的事,还用深度和法线当控制条件。做 3D 电商展示或虚拟人内容的朋友可以看看思路。

一篇新论文提出 GS-VTON-MVT,针对 3D Gaussian Splatting(3DGS)人物资产的服装替换问题,在 3D-VTONBench 基准上于 FID、KID、LPIPS 跨视角一致性及用户偏好等指标上超过现有通用 3D 编辑器和此前 3D 试穿方法。

正文

该框架分三个阶段:多视角修复 Transformer 用全跨视角自注意力联合编辑全部训练视图,让跨视角对应关系通过学习获得而非手工设定;几何引导多模态 ControlNet 把服装图像与当前 3DGS 状态渲染出的深度图、表面法线一起作为扩散编辑器的条件,在视觉证据最弱处补充显式结构;姿态自适应的 persona-aware attention 则依据相机姿态接近程度动态计算当前视图与参考视图特征的混合权重,替代固定权重。

此前流水线对各训练视图独立编辑,再靠人工挑选参考视图、固定注意力混合权重来融合,导致纹理漂移、印花模糊和新视角几何失真等问题被固化进 3D 表示。GS-VTON-MVT 将隐式与显式 3D 推理结合,对电商、虚拟形象和 3D 内容制作团队有直接应用价值。

原文 · arXiv cs.AI

Multi-View Transformer with Geometry-Guided Multi-Modal ControlNet for 3D Virtual Try-On

Editing a 3D Gaussian Splatting (3DGS) capture of a person so that they wear a new garment requires every rendered view to agree on the same garment details. Existing 3D virtual try-on (VTON) pipelines, however, edit training views largely independently and fuse them with hand-designed heuristics such as manually chosen reference views and fixed attention-blending weights. The resulting inconsistencies are baked into the 3D representation and surface as texture drift, blurred prints, and implausible geometry in novel views. We present GS-VTON-MVT, a three-stage framework that combines implicit and explicit 3D reasoning for consistent 3DGS try-on. (i) A multi-view inpainting transformer jointly edits a set of training views with full cross-view self-attention, so that cross-view correspondence is learned rather than imposed. (ii) A geometry-guided multi-modal ControlNet conditions the diffusion editor on the garment image together with depth and surface normals rendered from the current 3DGS state, supplying explicit structure where visual evidence is weakest. (iii) A pose-adaptive persona-aware attention replaces the fixed blending weight between current-view and reference-view features with weights derived from camera-pose proximity. On the 3D-VTONBench benchmark, GS-VTON-MVT improves FID, KID, LPIPS-based cross-view consistency, and user preference over general-purpose 3D editors and prior 3D try-on methods.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:3DGS、虚拟试穿、ControlNet、Transformer,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Sarthak Batra, Armin Mustafa, Adrian Hilton),经 AITOP 多维评分管线结构化提炼。