论文

研究:RL 后训练对扩散模型的增益可由固定激活向量解释

RL Post-Training in Diffusion Transformers Is Activation Steering

精选理由

这篇论文很反直觉:用一组固定的激活向量就能复现 RL 微调九成左右的增益,参数量只有 LoRA 的六十五分之一。做扩散模型微调或评估的人应该看看,可能会改变你对 RL 后训练到底在学什么的判断。

一项 arXiv 研究提出,RL 后训练给扩散模型带来的提升,大部分可以用一组与输入无关的 steering 向量来复现。研究者在冻结的 SD3.5-M 基座模型的各个 block 输出上加固定向量,参数量仅 0.29M,比 RL 训练的 LoRA(18.8M)少 65 倍。

正文

这些向量通过拟合公开 RL-tuned checkpoint 的速度预测得到,目标函数中没有奖励信号。虽然向量对所有 prompt、token 和采样 seed 都相同,但在六个 RL-tuned 图像模型上恢复了 90.0%–96.3% 的奖励增益,在两个视频模型上分别达到 89.8% 和 111.9%。将向量从零初始化并用在线 RL 训练后,仅凭向量就在 SD3.5-M 的文字渲染(OCR)和人类偏好(PickScore)上超过了基于权重的 RL。

作者据此推断,policy-gradient 后训练的大部分增益来自将预训练生成器导向其已有能力,而非学习新的输入相关变换。这一结论提示 RL 微调的效果空间可能远小于当前参数量所暗示的规模,对下游微调的成本估算有直接参考意义。

原文 · arXiv cs.LG

RL Post-Training in Diffusion Transformers Is Activation Steering

Reinforcement learning (RL) post-training substantially improves the visual quality, text rendering, and compositional accuracy of diffusion transformers. Because capabilities such as rendering requested text or composing multiple objects are inherently prompt-dependent, a natural hypothesis is that improving them requires learning complex, input-dependent transformations. To isolate how much of these gains an input-independent shift explains, we probe each RL-tuned model with a deliberately simple parameterization: a fixed set of steering vectors added to the block outputs of the frozen base model. On SD3.5-M, these vectors hold 65x fewer parameters than the RL-tuned LoRA (0.29M against 18.8M). We fit them to match the velocity predictions of public RL-tuned checkpoints, with no reward in the objective. Although the vectors are identical for every prompt, token, and sampling seed, they recover 90.0-96.3% of the reward gains of six RL-tuned image models, and 89.8% and 111.9% on two RL-tuned video models. Initialized at zero and trained with online RL, the vectors alone exceed the reward gains of weight-based RL on text rendering (OCR) and human preference (PickScore) on SD3.5-M. A constant shift thus suffices for most of the gain of policy-gradient RL post-training: without new input-dependent transformations, it steers the pretrained generator toward what it can already compute.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:SD3.5-M、扩散模型、强化学习、LoRA,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Tuna Han Salih Meral, Kaan Oktay, Alec Helbling, Atakan Kara, Duen Horng Chau, Pinar Yanardag),经 AITOP 多维评分管线结构化提炼。