Bradley-Terry 模型下的差分隐私排名统计推断方法
Differentially Private Statistical Inference for Rankings from Pairwise Comparisons
做 LLM 评测榜单的人可以看看这篇：在 Bradley-Terry 模型上加边级差分隐私，不只给排名，还给每个名次的置信区间和 top-K 检验，模拟实验和创意写作排名案例都有。要保护人类评委偏好数据时很实用。
一篇 arXiv 论文提出在边级差分隐私（edge differential privacy）约束下，基于人类成对比较对 LLM 排名做统计推断的方法，核心模型为 Bradley-Terry。作者对目标扰动估计量给出留一法（leave-one-out）分析，建立了欧氏误差和逐元素误差的有限样本上界，并在比较图足够稠密时把误差界推广到更小的隐私预算区间。方法还给出单个分数与分数差的渐近有效置信区间，并通过 plug-in ridge 偏差校正加 Gaussian-Laplace 校准改善有限样本表现。
针对排名本身，论文设计了 bootstrap 程序做同步排名推断，可输出单个排名的置信集合以及 top-K 成员资格检验。当比较概率公开且已知时，所有推断步骤不消耗额外隐私预算。作者用模拟实验验证有限样本性能，并把方法实际应用于对 LLM 创意写作能力的排名。对需要发布 LLM 榜单又不能泄露评委个体偏好数据的团队，这套工具提供了带置信区间的隐私保护排名方案。
Differentially Private Statistical Inference for Rankings from Pairwise Comparisons
Reliable inference for rankings of large language models based on human pairwise comparisons must preserve the privacy of individual preference records. To meet this need, we develop methods for score and rank inference under the Bradley-Terry model with edge differential privacy. For an objective-perturbed estimator, a new leave-one-out analysis establishes finite-sample bounds on Euclidean and entrywise estimation errors. For sufficiently dense comparison graphs, the error bounds extend to smaller privacy budgets, including regimes in which its privacy term dominates its statistical term. A uniform asymptotic expansion yields asymptotically valid confidence intervals for individual scores and pairwise score differences. To improve finite-sample performance, we account for both regularization bias and privacy-induced randomness by combining a plug-in ridge-bias correction with Gaussian-Laplace calibration. We further establish a bootstrap procedure for simultaneous rank inference, providing asymptotically valid confidence sets for individual ranks and tests of top-$K$ membership. When the comparison probability is public and known, all proposed inferential procedures require no additional privacy budget. We evaluate finite-sample performance through simulations and apply the proposed methods to rank large language models for creative writing.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：Bradley-Terry、差分隐私、LLM评测、统计推断，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.LG」（@Xintao Xia, Jia Gu），经 AITOP 多维评分管线结构化提炼。
