PORT：基于 token 级重加权的噪声偏好优化框架
PORT: Preference Optimization via Robust Token-Level Reweighting
训练时偏好数据常有标注噪声，这套方法能在 token 级别识别并压低被污染 token 的影响，而且只加一次前向传播，训练成本几乎不变。做 RLHF 或 DPO 微调的团队可以直接用他们开源的代码试试，和序列级方法比抗噪能力更强。
俄亥俄州立大学研究团队提出 PORT（Preference Optimization via Robust Token-level reweighting），用于在偏好标注存在噪声时改进 LLM 对齐效果。现有鲁棒偏好优化方法主要在序列层面操作，将整条回复视为一致正确或错误，但被拒绝的回复仍可能包含有价值的推理步骤，被偏好的回复也可能含有细微错误。
PORT 的核心机制是在 token 级别进行重加权：利用 token 对数概率的经验累积分布函数（CDF），构建一个只需前向传播的代理指标来近似梯度范数惩罚，从而选择性地抑制被噪声污染的 token，不增加反向传播的计算开销。理论上，作者证明 PORT 通过最小化噪声偏好标签引起的梯度偏差项的上界来降低梯度偏差。
在多种噪声设置下的实验中，PORT 的一致性和鲁棒性均优于现有序列级基线方法。代码已在 GitHub（osu-srml/PORT）开源，可复现其偏好对齐训练流程。
PORT: Preference Optimization via Robust Token-Level Reweighting
Preference optimization has become a central approach for aligning large language models with human values, but its effectiveness depends heavily on the quality of preference annotations. In practice, preference data is often noisy due to annotation errors and ambiguity. Existing robust preference optimization methods primarily operate at the sequence level, implicitly treating entire responses as uniformly correct or incorrect. However, rejected responses may still contain informative reasoning steps, while preferred responses can include subtle errors. In this work, we propose \textbf{P}reference \textbf{O}ptimization via \textbf{R}obust \textbf{T}oken-level reweighting \textbf{(PORT)}, a fine-grained framework for robust alignment under noisy preferences. PORT performs token-level reweighting using the empirical cumulative distribution function (CDF) of token log-probabilities, yielding an efficient forward-pass-only proxy to gradient-norm penalization that selectively suppresses corrupted tokens without additional backward-pass computation. We provide theoretical analysis showing that PORT reduces the gradient bias from noisy preference labels by minimizing an upper bound on the bias term. Extensive experiments across diverse noise settings demonstrate that PORT consistently improves robustness and outperforms existing sequence-level baselines. The code is available at https://github.com/osu-srml/PORT
本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：偏好优化、PORT、RLHF、DPO，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.AI」（@Ding Zhu, Xiukun Wei, Tian Xie, Zhihui Zhu, Xueru Zhang, Mohammad Mahdi Khalili），经 AITOP 多维评分管线结构化提炼。