arXiv 论文提出 Funnel-Aware Recalibration,修复漏斗下游概率失准问题
Lost in the Funnel: Selection-Induced Miscalibration in Ranking and Decision Systems
做搜索、广告或推荐系统的工程师都该看看这篇。它在生产环境 A/B 测试里把决策时点的 ECE 降了 60-80%,问题根源不是模型误差而是漏斗各环节的选择效应。如果你只用模型输出端验证校准,下游竞价和过滤可能一直在失真状态下做决策。
论文《Lost in the Funnel: Selection-Induced Miscalibration in Ranking and Decision Systems》研究排序与决策系统中分数在漏斗下游的校准失效问题。核心发现是:模型边界处已校准的分数经过过滤、重排、业务规则、竞价等环节的选择后,在校准失效的地方继续被当作真实概率使用。团队用逐层残差恒等式做诊断,以 rubric 判定的相关性为度量,可在每个漏斗层测量校准程度,并追踪到失准源于 selection 而非模型误差。提出的 Funnel-Aware Recalibration(FAR)在分数驱动决策的边界处做后处理重校准,且感知选择策略。
在双区域生产环境 A/B 测试中,FAR 将决策时点的期望校准误差(ECE)降低 60-80%,同时提升竞价效率。团队还改造两个公开数据集模拟行业场景验证该机制,并计划发布数据集及生成 pipeline。论文最后总结了多阶段打分系统校准的操作性原则。
与只关注 CTR 等表现依赖指标的做法不同,该工作把校准验证延伸到分数被实际使用的环节,对搜索、广告、推荐等组合多个组件的系统的工程实践有直接指导价值。
Lost in the Funnel: Selection-Induced Miscalibration in Ranking and Decision Systems
Modern ranking and decision systems combine components with divergent objectives, and scores calibrated at the model boundary are used as calibrated probabilities throughout the funnel, across filtering, reranking, business-rule enforcement, and auctions. Yet calibration is typically validated only where the score is produced, so its breakdown where the score is used is routinely overlooked. We ask when calibration survives this selection, how inherited error differs from selection effects, and what can fail for a perfectly calibrated score at a component boundary. We study these questions through a boundary-wise diagnostic framework of residual identities. We target rubric-judged relevance rather than presentation-dependent metrics such as CTR, so calibration is measurable at every funnel layer of interest. We first present an industry-wide case study that applies our diagnosis to settings where calibration deteriorates at decision time, tracing the degradation to selection rather than model error. This diagnosis motivates Funnel-Aware Recalibration (FAR): post-hoc, selection-policy-aware recalibration at the boundary where scores drive decisions. In a live two-region production A/B test, FAR cuts decision-time expected calibration error (ECE) by 60-80% and improves auction efficiency. We further adapt two public datasets to simulate industry settings, providing additional evidence for the mechanism. We plan to release the adapted datasets and their generation pipeline to support further research. Finally, we distill these findings into practical operating principles for calibrating multi-stage scoring systems.
本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:校准、排序系统、ECE、推荐系统,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Aditya Somani, Rahul Ragesh, Vikas Joshi, Amit Singh),经 AITOP 多维评分管线结构化提炼。