NOPO：只用负样本做强化学习，应对单侧可验证奖励场景
Negative-Only Policy Optimization for One-Sided Verifiable Rewards
做 RLVR 训练的同学可以看看：当验证器只能判错不能判对时，这篇论文的 NOPO 干脆只用负样本更新策略，避免奖励投机解法。在代码、数学、逻辑基准上 pass@k 都跑赢了标准 RLVR 基线。
arXiv 论文（编号 2611.17287）提出 Negative-Only Policy Optimization（NOPO），针对一类被作者定义为单侧可验证性（OSV）的场景：验证器只能可靠地判定错误答案，而通过的候选答案中可能混有大量假阳性。论文指出，标准 RLVR 在此场景下会直接强化那些钻过部分检查空子的错误解法。
NOPO 的做法是把验证通过的样本当作无标签数据，仅通过抑制可信的验证负样本来更新策略。方法由两部分组成：一个相对参考模型的负样本抑制损失，以及一个组级适配器，按未被判错响应的比例缩放每次更新。
在单元测试、约束检查和 LLM 自验证三类验证器下，NOPO 在代码、逻辑、数学基准上于无 oracle 奖励时 pass@k 超过基线，且增益在推理时扩展和跨基准迁移中保持。
Negative-Only Policy Optimization for One-Sided Verifiable Rewards
Reinforcement learning with verifiable rewards (RLVR) typically relies on a verifier that can reliably judge whether sampled solutions are correct. However, in many realistic settings, complete verification is generally unavailable, while cheap checks can only falsify failures. We formalize this regime as one-sided verifiability (OSV): verified negatives are reliable counterexamples, but verified positives are merely non-falsified candidates and may contain many false positives. This asymmetry makes standard RLVR unreliable, because reinforcing verified positives can directly reward shortcuts that pass partial checks. To address this issue, we propose Negative-Only Policy Optimization (NOPO), which treats verified positives as unlabeled and updates the policy only by suppressing trusted verified negatives. Specifically, NOPO combines a reference-relative negative suppression loss with a group-level adapter that scales each update by the proportion of non-falsified responses. Across unit-test, constraint-check, and LLM self-verifiers, NOPO improves pass@k over baselines on code, logic, and math benchmarks when oracle rewards are unavailable. These gains persist under inference-time scaling and cross-benchmark transfer.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：NOPO、RLVR、强化学习、论文，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.LG」（@Jiacheng Xu, Shuo He, Fuxiang Zhang, Chaojie Wang, Bo An），经 AITOP 多维评分管线结构化提炼。