Selective Tie Learning：用平局样本减少偏好优化中的伪相关
Selective Tie Learning: Mitigating Spurious Correlations in Preference Optimization
训练偏好模型总学到不该学的关联？这篇论文用平局样本解决了，还告诉你怎么挑最有用的那些，实验覆盖到大语言模型。
一项 arXiv 研究提出 Selective Tie Learning 方法，针对偏好优化易学到伪相关的问题，在训练数据中引入平局样本（equally preferred responses）。研究推导出每个平局样本对伪相关依赖的影响函数，证明在同等数量子集中选取预测降幅最大的样本是一阶最优策略。由于真实影响计算需要伪特征信息和目标函数曲率，作者用隐式奖励边际幅度作为代理指标，并结合广义 Gauss-Newton 近似来估计曲率。在 log-linear 策略、神经网络模型和大语言模型上的实验显示，该方法比均匀选取平局样本用更少数据就达到相当的伪相关降低效果。
Selective Tie Learning: Mitigating Spurious Correlations in Preference Optimization
Preference optimization can learn spurious correlations, and augmenting the preference data with ties (equally preferred responses) can reduce spurious reliance. However, ties differ in their corrective effect, making their selection for augmentation important. For log-linear policies, we derive each tie's influence on spurious reliance and show that taking the largest predicted reductions is first-order optimal among subsets of equal size. Computing these influences requires knowledge of the spurious features and the curvature of the training objective, which is often unavailable or expensive to estimate. To this end, we explore using the magnitude of a tie's implicit reward margin and characterize the conditions under which it is a useful proxy for predicted influence. To test the theory and its qualitative transfer to neural models, we estimate influence with a counterfactual proxy for spurious reliance and use a generalized Gauss--Newton approximation to curvature for language models. Across log-linear policies, neural models, and large language models, our selection rules consistently achieve comparable reductions in spurious reliance with fewer ties than uniform selection.
