UECR-GRPO：用熵校准统一策略内蒸馏与GRPO的信用分配
When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignment
教小模型做数学题的新训练方法，把教师模型的逐token提示和答案对错验证揉进一个GRPO更新里，Qwen3-1.7B和4B都涨了不到1个点，做RL训练的可以看看细节。
RLVR 只靠最终答案正确性给奖励，难以指导单个 token，而策略内蒸馏（OPD）的教师偏好不一定反映答案对错。论文提出 UECR-GRPO，把验证器信号和教师信号整合进同一个 GRPO 式更新，覆盖响应级和 token 级。其中 PUU 在组归一化和 PPO 裁剪前合并教师评分与验证器奖励，让教师证据参与响应排序；ECR 用教师与旧策略的 token 差值再分配奖励，并用全词表教师熵抑制不确定指导。在五个数学推理基准上，Qwen3-1.7B 和 Qwen3-4B 学生的 Avg@12 准确率分别达到 17.21% 和 65.09%，超过各自规模最强基线 0.89 和 0.56 个百分点。
When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignment
Reinforcement learning with verifiable rewards (RLVR) supervises mathematical reasoning through final-answer correctness, but provides little guidance on individual tokens. On-policy distillation (OPD) supplies dense feedback on student-generated responses, yet teacher preference need not reflect correctness. Recent hybrids combine OPD and verifier-derived advantages or reweight task credit using teacher ratios. However, teacher guidance enters after verifier-based group normalization, and token reweighting need not preserve the total task credit assigned to each response. We introduce Unified Entropy-Calibrated Credit Redistribution for GRPO (UECR-GRPO), which integrates verifier and teacher signals within a single GRPO-style update at both the response and token levels. \emph{Path-Utility Unification} (PUU) combines verifier reward and a teacher-to-anchor path log-ratio in a single KL-regularized objective. Its on-policy implementation uses a length-normalized teacher score and combines both rewards before group normalization and PPO clipping, allowing teacher evidence to influence the response ranking. \emph{Entropy-Calibrated Redistribution} (ECR) then uses the signed teacher--old-policy token gap to redistribute the verifier-derived component. Full-vocabulary teacher entropy attenuates uncertain guidance, while a response-wise zero-sum projection preserves the total task credit and its token-wise sign before clipping. Across five mathematical reasoning benchmarks, UECR-GRPO achieves average \(\mathrm{Avg@12}\) accuracies of 17.21\% and 65.09\% with Qwen3-1.7B and Qwen3-4B students, respectively, exceeding the strongest baseline at each scale by 0.89 and 0.56 percentage points.
