SetGRPO 提出校准评分器扰动的策略优化方法，在 Qwen3.5 与 Llama3.2 上十个基准拿下最高宏平均
SetGRPO: Calibrating Grader-Induced Policy Optimization
用评分器训开放式任务的 RL 方法容易因打分波动跑偏，这篇给 GRPO 加了不确定性和风险控制，实测在 Qwen3.5 和 Llama3.2 上十个基准平均分最高。
针对基于评分器的强化学习（GRPO）中评分随机性会改变整组训练系数的问题，论文提出 SetGRPO。方法包含三个模块：联合奖励集合校准、几何引导安全恢复和选择性风险门控优化，在可交换性条件下提供有限样本边际覆盖与风险控制。在覆盖三个领域的十个开放式基准上，SetGRPO 在 Qwen3.5 和 Llama3.2 两个模型家族的所有对比训练方法中取得最高宏平均。
SetGRPO: Calibrating Grader-Induced Policy Optimization
Rubric-based reinforcement learning uses fine-grained grader feedback to train large language models on open-ended tasks, but grader randomness can alter GRPO updates in ways that score discrepancies alone cannot characterize. Because GRPO jointly normalizes rewards within each response group and combines the resulting training coefficients with response-level gradients, a scoring discrepancy can shift the coefficients of the entire group and induce policy-dependent update changes. Selective execution further affects the direction-reversal risk of retained updates. To address this, we propose Set-Valued Group Relative Policy Optimization (SetGRPO). It comprises three modules: (a) Joint Reward Set Calibration constructs a joint uncertainty set for the training coefficients; (b) Geometry-Guided Safe Recovery maps candidate coefficients into the current policy's local update geometry and determines the largest safe recovery ratio for each response group; and (c) Selective Risk-Gated Optimization selects a global threshold through independent risk calibration to control the conditional direction-reversal risk of accepted updates while preserving the original GRPO objective. Under exchangeability, SetGRPO provides finite-sample marginal coverage for the joint coefficient set and simultaneous risk control across candidate thresholds. On ten open-ended benchmarks spanning three domains, SetGRPO achieves the highest macro average among all compared training methods on both Qwen3.5 and Llama3.2 model families.