HP-RLVR：单条轨迹完成 token 级信用分配，训练速度最高达 GRPO 2.30 倍
Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
训练 LLM 推理的同学看看：单条 rollout 就能做 token 级信用分配，数学上还比 GRPO 快 1.2 到 2.3 倍，代码比较可以少八倍采样。
论文提出 Hindsight-Privileged RLVR（HP-RLVR），用单条 rollout 替代 GRPO 的多采样组比较。方法对每个 token 计算因果视图与包含真实未来和验证器反馈的后见视图之间的归一化 Jensen-Shannon 散度，生成有界的 token 级更新权重，无需参考答案或 critic。在 Qwen3-8B 的 100 步配对实验中，HP-RLVR 在数学、代码生成和 ALFWorld 上分别比 GRPO 快 1.21 倍、1.46 倍和 2.30 倍，每 prompt 只用八分之一的 rollout。准确率整体相当，数学任务上 HP-RLVR 在三个尺寸全部领先；ALFWorld 中 GRPO 有 43.2%（4B）到 49.0%（8B）的梯度范数记录为零，HP-RLVR 则没有。
Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual reasoning tokens or actions remains difficult. Group-relative methods such as Group Relative Policy Optimization (GRPO) sample several attempts per prompt, but lose their reward-based learning signal when all attempts receive the same outcome. We propose Hindsight-Privileged RLVR (HP-RLVR), a single-rollout method that turns a completed trajectory's own future into token-level update weights. The generating policy snapshot scores each eligible token under its causal history and under a hindsight context containing the realized future and verifier feedback. A normalized Jensen--Shannon divergence between the two views weights the reward relative to a historical baseline, yielding bounded token credit for both successful and failed attempts, with no reference solution or learned critic. In paired Qwen3-8B runs over a shared $100$-step schedule, HP-RLVR trains $1.21\times$, $1.46\times$, and $2.30\times$ faster than GRPO on mathematics, code generation, and ALFWorld, using one eighth the rollouts per prompt. Accuracy is comparable overall, with HP-RLVR ahead on mathematics at all three sizes and GRPO ahead on code and ALFWorld. In ALFWorld, $43.2\%$ (4B) and $49.0\%$ (8B) of GRPO's gradient-norm records are zero, versus none for HP-RLVR.
