选择性预测何时能被认证?论文用动态规划权衡安全、粒度与流量
Available Guardrails: Certifying Selective Prediction across ML Systems
arXiv 论文把"安全闸门能否过认证"变成动态规划问题,还能算校准数据怎么分,做部署的可翻翻。
一篇 arXiv 论文把选择性预测的"可认证性"变成可计算的量:用精确二项反演判断有限校准数据能否在目标精度下为每个报告单元(工具、策略标签、患者亚组)颁发证书。在固定组顺序下,报告分区选择被形式化为动态规划,暴露出安全、粒度与已服务流量之间的权衡前沿。结果显示,掌握真实信息的规划者可比 support balancing 高出 0.157 的平均覆盖率,而朴素估计只能恢复 0.005,有限数据下的恢复成为核心难题。在一个划分上构造候选、在另一个上选择的方法将平均覆盖率提升 0.060,该方向在三个意图路由数据集、两种架构的 60 个模型效应中复现了 59 次。同一前沿在 LLM 工具调用、内容审核、病灶分类和推荐系统中以各自上限重复出现。
Available Guardrails: Certifying Selective Prediction across ML Systems
A selective predictor acts as a safety gate: it returns an output only when the prediction appears sufficiently trustworthy. Deployments increasingly require this reliability to be certified at a target precision for every reporting unit of interest, such as a tool, policy label, or patient subgroup. The main difficulty is often not whether a granted certificate is valid, but whether finite calibration data can produce one at all. As the gate becomes safer or more fine-grained, some units may receive too little evidence to certify. We make this notion of availability computable through classical exact-binomial inversion and formulate reporting-partition selection, under a fixed group order, as a dynamic program that exposes the trade-off among safety, granularity, and served traffic. The resulting frontier reveals a large population opportunity that finite-sample estimation nearly erases: a truth-informed planner gains $0.157$ mean coverage over support balancing, whereas a naive estimator recovers only $0.005$, making recovery from finite data the central challenge. Constructing candidate partitions on one planning split and selecting among them on another recovers part of this gap, improving mean coverage over support balancing by $0.060$, with the direction reproduced in $59$ of $60$ model effects across three intent-routing datasets and two architectures. A complementary validity-preserving lever, reallocating the familywise error budget across reporting units, recovers additional coverage both with population quantities and noisy estimates. The same frontier recurs, with predictor-specific ceilings, across LLM tool-calling, content moderation, lesion classification, and recommendation. Certified availability is therefore a plannable deployment resource that determines when a safety gate can be certified, at what granularity, and over how much traffic.