差分隐私下的稀疏 RLHF 奖励估计理论框架
Differentially Private Sparse Reward Estimation with Preference Feedback
做 RLHF 隐私保护方向的可以看：这篇给了稀疏奖励估计的首套 DP 理论和算法，误差界只跟稀疏度 s* 挂钩，不随整个模型维度膨胀。
这篇论文研究 Private Sparse RLHF 问题，假设奖励模型只由 s* 个远小于模型维度 d 的相关特征决定。作者在 Bradley-Terry-Luce 模型下给出了首个差分隐私稀疏奖励估计的完整理论框架，针对 Sample-DP 与 Label-DP 两种设置设计了高效算法。误差上界分别为 O~(s*/n + s*²/n²ε²)（Sample-DP）和 O~(s*/n + s*/nε²)（Label-DP，基于 randomized response），并证明 Sample-DP 速率接近最优。在情感生成与安全对齐基准上，该方法比稠密基线表现出更好的隐私-效用权衡。
Differentially Private Sparse Reward Estimation with Preference Feedback
Reinforcement Learning with Human Feedback (RLHF) has become a central paradigm for aligning Large Language Models with human values. However, the reliance on sensitive preference data necessitates rigorous differential privacy (DP) guarantees to prevent user information leakage. Existing private alignment methods typically assume dense reward parameterizations, resulting in estimation error bounds that scale polynomially with the ambient {model size $d$}. Such dependence is prohibitive in modern {overparameterized models} where $d$ is vast. In this work, we study \emph{Private Sparse RLHF} problem to address this limitation, assuming that human preferences (reward model) are governed by a subset of $s^* \ll d$ relevant features. We provide the first comprehensive theoretical framework for differentially private sparse reward estimation from pairwise preference feedback under the Bradley-Terry-Luce model, and introduce efficient algorithms for both Sample-DP and Label-DP settings. Theoretically, under the squared $\ell_2$ parameter estimation error, we establish upper bounds of $\widetilde{O}(s^*/n + {s^*}^2/n^2\varepsilon^2)$ for $(\varepsilon,δ)$-Sample-DP and $\widetilde{O}(s^*/n + s^*/n\varepsilon^2)$ for $\varepsilon$-Label-DP based on randomized response, where $n$ is the training data size. For the lower bounds, we show that the Sample-DP rate is nearly optimal, and that the guarantee for our Label-DP method cannot be further improved within the randomized-response framework. We also provide a general lower bound for the problem with Label-DP. Empirically, our framework demonstrates superior efficacy over dense baselines across sentiment generation and safety alignment benchmarks, offering a significantly more favorable privacy-utility trade-off.
