论文

ROAM-GRPO:通过离模式策略挖掘提升 GRPO 推理多样性

ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining

精选理由

做推理模型 RL 后训练的人可以看看这篇。它给 GRPO 加了个离模式挖掘机制,专治 SFT+RLVR 后输出千篇一律、pass@k 不涨的问题。在 MATH lvl5 上用 Qwen3.5-2b 验证,pass@16 提了 4.7%,多样性指标翻倍,方法可直接复现。

一篇新论文提出 ROAM-GRPO(Routed Off-mode Approach Mining for GRPO),针对 SFT 加 RLVR 后训练流程导致策略模式坍缩、输出同质化的问题,对 GRPO 算法进行增强,鼓励模型探索不同的推理策略。

正文

机制上,ROAM-GRPO 在训练中检测已被很好解决的问题,将其转入“离模式”通道,提示模型生成不同的解题策略,由 LLM judge 验证逻辑有效性与差异性,再自蒸馏回策略网络。实验在 MATH lvl5 数据集的 1000 题子集上训练 Qwen3.5-2b,峰值 pass@16 提升 4.7%,正确输出的嵌入余弦距离提升约 1.5-2 倍。

对做推理模型后训练的团队来说,这项工作给出了缓解 pass@k 精度收敛或退化、提升输出多样性的具体可复现方案。

原文 · arXiv cs.AI

ROAM-GRPO: Enforcing Reasoning Diversity in GRPO via Routed Off-mode Approach Mining

Supervised fine-tuning (SFT) paired with Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant recipe for post-training LLMs on complex reasoning tasks, but has been shown to collapse the policy into a low-diversity mode. Repeated LLM outputs are nearly identical, and pass@k accuracy converges or degrades as a result. We propose ROAM-GRPO (Routed Off-mode Approach Mining for GRPO), an augmentation to the GRPO algorithm which incentivizes exploration of distinct reasoning strategies and aims to increase output creativity during evaluation. ROAM-GRPO detects well-solved problems during training and transfers them to an "off-mode" track. The model is prompted to produce distinct solution strategies to such problems, which are verified for logical validity and distinctness by an LLM judge and subsequently self-distilled into the policy. Training Qwen3.5-2b on a 1,000-problem subset of the MATH lvl5 dataset, ROAM-GRPO implementation increases peak pass@16 by 4.7% and correct embedding cosine distance by approximately 1.5-2x on a difficult evaluation set.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:GRPO、Qwen3.5-2b、RLVR、推理模型,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Jeff Duan, Matt Thomson),经 AITOP 多维评分管线结构化提炼。