Advantage Guidance：用监督学习的稳定性做强化学习
Reinforcement Learning via Advantage Guidance
Google DeepMind 这篇论文把监督学习和强化学习的好处合在一起，26 个稀疏奖励任务上都跑赢了以前的 RL 方法，搞 RL 的可以看看。
arXiv 论文 2610.19783 提出新策略提取方法 Advantage Guidance。它在监督学习的训练结构上加入优势条件（advantage conditioning），并在采样阶段通过对比扩散引导（contrastive diffusion guidance）逼近 KL 正则化的最优策略。该方法在 26 个长时程、稀疏奖励任务上显著超过此前的 state-of-the-art RL 方法，覆盖了不同最优程度的数据集。论文针对的是 RL 常见的训练不稳定和超参数敏感问题。
Reinforcement Learning via Advantage Guidance
Reinforcement learning (RL) holds the promise of learning a policy that can outperform what is shown in the data. However, in practice, such systems suffer from instability and hyperparameter sensitivity. In contrast, supervised learning provides a stable and scalable training objective, but is fundamentally limited by the quality of the demonstrated behavior. In this paper, we ask whether we can perform policy optimization with something as simple and stable as supervised learning but as performant as RL. Unfortunately, naively training the policy with supervised learning and conditioning on future rewards or advantages typically underperform dedicated RL methods. We introduce a new policy extraction method, Advantage Guidance, that retains the simple supervised learning structure with advantage conditioning while outperforming prior RL methods through effective conditional training and contrastive diffusion guidance during sampling. We show that Advantage Guidance can significantly improve a model's ability to respond to conditional advantage signals, and the contrastive guidance during sampling can use the learned advantage-conditioned policy to target the KL-regularized optimal policy. Empirically, we find that Advantage Guidance significantly outperforms prior state-of-the-art RL methods on 26 long-horizon, sparse reward tasks across datasets with varying degrees of optimality.
