RubricOpt：让模型在偏好优化中自我修订语义评分标准
RubricOpt: Self-Revision of Semantic Rubrics for Preference Optimization
这篇论文让模型自己改训练用的评分标准，再用改好的 rubric 做 DPO。结果很诚实：有的提升有的回退，适合做偏好优化的人看看细节。
arXiv 论文 RubricOpt 提出一个框架，让策略模型在 Direct Preference Optimization（DPO）训练中自行修订行为准则。模型先诊断自己的无 rubric 输出，保留已被覆盖的要求，再编辑导致反复失败的条款。共享语义表征连接任务需求、偏好数据对比和 rubric 条款，并由确定性门控检查证据引用与固定语义锚点。在能力与开放式生成基准上的实验显示，rubric 条件化优于普通 DPO，但增益和回退因模型和数据集而异。
RubricOpt: Self-Revision of Semantic Rubrics for Preference Optimization
A model's failures can reveal gaps in the behavioral criteria guiding its training. We introduce RubricOpt, a framework that lets a policy revise these criteria within preference optimization. The policy diagnoses its own rubric-free outputs, retains requirements already covered by the rubric, and edits clauses that leave recurring failures unaddressed. A shared semantic representation connects task demands, preference-data contrasts, and rubric clauses, while a deterministic gate checks evidence references and fixed semantic anchors. The rubric then guides Direct Preference Optimization (DPO) during training, with preference pairs, labels, and the objective held fixed, and is removed at evaluation. Our analysis relates the local effect on rubric-free success to semantic alignment and a residual beyond the measured factors. Experiments on capability and open-ended generation benchmarks show that rubric conditioning can improve outcomes over ordinary DPO, with gains and regressions varying across models and datasets. A matched continuation isolates the contribution of revision from additional training and finds mixed incremental effects for the tested edit, whose feedback overlaps the evaluation set. RubricOpt thus brings the behavioral specification itself into the policy's feedback loop.