Tencent Hunyuan 研究批大小扩展理论，PPO 生成吞吐提升至 2.29 倍
腾讯混元把经典批大小理论搬进了 LLM 强化学习，实测 PPO 吞吐提升 2.29 倍、GRPO 训练省 29% 时间，做 RL 训练的可以看看怎么调参省钱。
Tencent Hunyuan 发布新研究，将经典的 critical-batch-size 理论扩展到在线 LLM 强化学习场景，覆盖 GRPO 和 PPO 两种算法。研究发现学习率重调可以在一定批大小范围内保持每条响应的学习效果。在固定硬件上，扩大批大小让 PPO 生成阶段吞吐最高提升 2.29 倍。最优 GRPO 配置达到相同验证目标所需时间减少 29%。
⚡️ As LLM reinforcement learning scales to larger GPU clusters and more training data, training efficiency becomes a first-order concern. Our new research revisits classical critical-batch-size theory and extends it to online LLM RL, where the model generates its own training data and rollout generation and training scale differently. Across GRPO and PPO, we find that learning-rate retuning can preserve learning per response over a bounded range of batch sizes. On fixed hardware, scaling up the batch size improves PPO generation-stage throughput by up to 2.29×, while our best measured GRPO configuration reaches the same validation target in 29% less time. 🚀 Read the full research: hy-ai.woa.com/research/10011… 5 💬 0 🔄 1 ❤️ 20 👀 836 📊 3 ⚡
