DARS：用密度感知奖励缩放解决RL模式坍缩问题
Density-Aware Reward Scaling: A Reinforcement Learning Framework for Reward-Proportional Sampling
这篇论文给 GRPO 打了个小补丁：按结果密度反着缩放奖励，就能不牺牲对齐的前提下让文生图多样很多，改动量很小。
arXiv 论文提出 Density-Aware Reward Scaling（DARS），一种奖励处理机制，用于在连续结果空间实现奖励成比例采样。DARS 估计每个 on-policy rollout 组内结果的局部密度，并按密度反向缩放奖励，不需要学习密度模型、辅助目标、critic 或修改策略梯度目标。在多模态导航任务中，DARS-GRPO 避免了 GRPO 出现的单模式坍缩，并接近恢复目标模式分配。应用于文本到图像提示自适应时，DARS-GRPO 在多个数据集、Policy LM 和奖励设置下一致提升图像与提示多样性，同时保持语义对齐。
Density-Aware Reward Scaling: A Reinforcement Learning Framework for Reward-Proportional Sampling
Reinforcement learning objectives that maximize expected return can concentrate probability mass on a small subset of high-reward outcomes, limiting diversity when multiple distinct solutions are desirable. We introduce Density-Aware Reward Scaling (DARS), a simple reward-processing mechanism that promotes reward-proportional sampling in continuous outcome spaces while retaining standard policy-gradient training. DARS estimates the local density of outcomes within each on-policy rollout group and inversely scales rewards by this density. The method requires no learned density model, auxiliary objective, critic, or modification to the underlying policy-gradient objective. In a controlled multimodal navigation task, DARS-GRPO avoids the single-mode collapse exhibited by GRPO and closely recovers the target mode allocation. Applied to text-to-image prompt adaptation, DARS-GRPO consistently improves image and prompt diversity across datasets, Policy LMs, and reward settings while maintaining good semantic alignment. These results show that density-aware reward scaling provides a lightweight way to improve coverage of diverse high-reward outcomes within existing reinforcement-learning pipelines.