PICO 方法解决 RLVR 在难题上全部 rollout 失败的困境
When All Rollouts Fail: Policy-Compatible Joint Decoding for Hard-Problem Reasoning
遇到难题所有 rollout 都失败时 GRPO 就学不到东西，这篇论文的 PICO 方法让专家轨迹能被策略兼容地利用，做 RL 训练的可以看看。
RLVR（如 GRPO、RLOO）在难题上有一个学习悬崖：当所有 rollout 都失败时，组相对估计器会把优势更新归零，模型学不到任何东西。论文提出 PICO（Policy-Compatible Reinforcement Learning），通过分布约束的联合解码让专家数据在 token 级引导解码，同时用动态 top-k 机制把采样分布限制在有限散度预算内，保证轨迹既可验证又与当前策略兼容。这些轨迹与 on-policy rollout 在混合行为目标下用重要性采样联合优化。实验覆盖多个推理数据集和智能体基准，PICO 持续优于标准 RLVR 和既有 off-policy 基线。
When All Rollouts Fail: Policy-Compatible Joint Decoding for Hard-Problem Reasoning
Reinforcement Learning with Verifiable Rewards (RLVR) derives its learning signal from binary rewards via on-policy exploration and becomes a powerful paradigm for improving the reasoning of large language models. Yet it faces a learning cliff on problems beyond model's current ability: when every rollout fails, the samples reveal nothing about a correct solution, and group-relative estimators (e.g., GRPO, RLOO) collapse the update to zero advantage. Expert trajectories offer a remedy, but they are incompatible with the current policy and often lie in the student's low-probability regions, so the surviving updates demand large parameter shifts and decay into surface-level imitation rather than transferable reasoning. We argue that a useful learning signal on hard problems must be both verifiably correct and sufficiently compatible with the current policy. We propose Policy-Compatible Reinforcement Learning (PICO), which performs distribution-constrained joint decoding: expert data guides decoding at the token level while preserving the student's relative token probabilities, and a dynamic top-k mechanism confines the sampling distribution under a finite divergence budget, yielding trajectories that are verified and policy-compatible by construction. These trajectories are optimized jointly with on-policy rollouts under a mixed-behavior objective that mitigates their distribution mismatch via importance sampling. Experiments across reasoning diverse datasets and agentic benchmarks demonstrate that PICO consistently outperforms standard RLVR and prior off-policy baselines.
