NVIDIA 推出 LoGRA：显存占用低于 LoRA 的后训练 RL 方法
显存不够跑 RL 的朋友看过来，NVIDIA 这个 LoGRA 比用 LoRA 还省显存，7B 模型省了近一半，精度基本不掉。
NVIDIA 发布 LoGRA 论文，一种面向 LLM 后训练 RL 的省显存技术。在 1.5B 模型上，LoGRA 平均训练显存 7.2 GiB，低于 rank-256 LoRA 的 13.2 GiB，Pass @4 两者接近（81.8% 对 81.4%）。其原理是用每步重绘的 rank-256 随机投影压缩权重梯度，不构建完整梯度，并用 KL 预算约束策略更新幅度。在 7B 模型上，每 GPU 平均训练显存下降 45.7%（31.8 降至 17.3 GiB），Pass @1 精度 72.3% 接近 dense Adam 的 72.5%。该方法还可在单个八 GPU 节点上对 27B 模型训练超过 1,100 步，而 dense Adam 在此配置下会显存不足。
Recommended NVIDIA paper on memory-efficient post-training RL. A very interesting technique for scaling RL for LLMs. The main takeaway is that you can update all of a model's weights with RL and still use less memory than LoRA. At 1.5B, LoGRA averages 7.2 GiB of training memory, while rank-256 LoRA needs 13.2 GiB. Pass @4 is about the same (81.8% and 81.4%), and Pass @1 is about 2 points lower. If GPU memory is what stops you from running RL on a bigger model, this method is worth a look. LoGRA never builds a full gradient for the weight matrices it compresses. As microbatches accumulate, each weight gradient is compressed with a rank-256 random projection that is redrawn every step, and the update is rebuilt from two small matrices. Before each step, it predicts how much the policy will change and scales the step to stay within a KL budget. Average training memory per GPU drops 45.7% at 7B (31.8 to 17.3 GiB), with peak accuracy close to dense Adam (72.3% and 72.5% Pass @1 ). It also trains a 27B model for more than 1,100 steps on a single eight-GPU node, where dense Adam runs out of memory. Paper: arxiv.org/abs/2610.06647 Chat with Paper: academy.dair.ai/papers/logra-s… 💬 3 🔄 2 ❤️ 11 👀 729 📊 6 ⚡
