论文解析：DPO 稀疏更新的理论解释——NTK 视角下的双重稀疏性
On the Sparsity of Direct Preference Optimization: Weight Disentanglement in the NTK Regime
如果你调过 DPO 却纳闷为什么只动了一点参数就有效，这篇论文给了理论答案，还顺带解释了为什么 DPO 后的模型更好合并。
arXiv 论文《On the Sparsity of Direct Preference Optimization》用神经正切核（NTK）线性化分析 DPO 的优化动态，发现其 logistic 损失结构带来隐式最大间隔偏置。论文提出 dual sparsity 概念：数据层面由支持向量驱动，参数层面来自偏好与非偏好响应之间的梯度抵消。主要定理用依赖间隔的谱量界定了支持对的数量，并将双重稀疏性与利于模型合并的权重解耦联系起来。实验覆盖 GPT-2 Small/Medium 以及 Llama-3-8B、Mistral-7B、Gemma-2-9B，使用 Anthropic HH-RLHF 和 UltraFeedback 数据集；结果显示 DPO 相比监督微调可将跨任务干扰降低至多 35 倍。
On the Sparsity of Direct Preference Optimization: Weight Disentanglement in the NTK Regime
Direct Preference Optimization (DPO) has been widely adopted for aligning language models with human preferences. Recent empirical work has revealed that DPO induces remarkably sparse parameter updates, yet no theoretical explanation for this phenomenon has been offered. We analyze DPO's optimization dynamics under a neural tangent kernel linearization and show that the logistic structure of the DPO loss gives rise to an implicit maximum-margin bias. Building on this observation, we introduce the concept of dual sparsity: the DPO update direction is a sparse linear combination (data-level sparsity, driven by support vectors) of sparse gradient feature vectors (parameter-level sparsity, arising from gradient cancellation between preferred and dispreferred responses). We formalize this structure in our main theorem, bound the number of support pairs by a margin-dependent spectral quantity, and present a framework relating dual sparsity to weight disentanglement---a condition favorable for model merging. Experiments on GPT-2 Small and Medium and on 7--9B instruction-tuned models (Llama-3-8B, Mistral-7B, Gemma-2-9B) with the Anthropic HH-RLHF and UltraFeedback datasets confirm both sources of sparsity, and experiments on GPT-2 show that DPO reduces cross-task interference by up to $35{\times}$ compared to supervised fine-tuning.