Routing Beyond Task Categories: 基于 LAB-Bench 的成本感知路由研究
Routing Beyond Task Categories: A Cost-Aware LAB-Bench Study
想在有限预算里用便宜模型跑、必要时切贵模型的人应该看看这篇。它在 LAB-Bench 上实测了几种路由策略,发现语义路由和 deferral 各提 0.66 和 1.35 个百分点,但置信区间跨零,效果未必稳。写模型路由代码前值得对齐一下预期。
这项研究在 LAB-Bench 上测试路由策略能否在固定推理预算内提升准确率。试点阶段,基于题目类别的路由器和增加六个结构化特征的路由器都未达到 80% 主目标,在 120 道留出题上两者都选择 Haiku 并得 71/120。
冻结方案后在 144 道新题上测试:语义路由得 86/144,允许读完 Haiku 答案后再推迟(deferral)的策略得 87/144,两者均控制在每千次查询 0.678 美元预算内。
相比 95% Haiku + 5% GPT 的混合基线,语义路由带来 0.66 个百分点提升,deferral 带来 1.
35 个百分点,配对 95% 置信区间分别为 [-0.28, 2.
25] 和 [-0.14, 3.
47]。一条对 Haiku 非答案直接升级的简单规则得 92/144,但成本升至每千次 0.
710 美元,超出预算。后续开发检查中 Astra 和 Gemini 各得 30/40,各路由家族最佳策略也达到该分数,但未达到 32/40 标准,原定的 120 题评估未启动。
Routing Beyond Task Categories: A Cost-Aware LAB-Bench Study
We tested whether features of biology questions or of a model's first answer can improve accuracy within an inference budget. In a LAB-Bench pilot, we compared a category router with one that adds six structural features. Neither reached the primary 80% accuracy target on development. At the 50% and 60% secondary targets, both chose Haiku for all 120 held-out questions and scored 71/120. We then froze a new comparison before collecting responses on 144 additional questions. Semantic routing scored 86/144. A policy that could defer after reading Haiku's answer scored 87/144. Both stayed within the inference budget of USD 0.678 per 1,000 queries. Against a 95% Haiku and 5% GPT mixture, the accuracy gains were 0.66 percentage points for semantic routing and 1.35 points for deferral. The paired 95% intervals were [-0.28,2.25] and [-0.14,3.47] points. A simple rule that escalated Haiku nonanswers scored 92/144 at USD 0.710 per 1,000 queries, above the budget. A retrospective nonanswer rule with a calibration-selected escalation probability gave an expected 85.63/144. At the learned deferral policy's observed inference cost, post hoc interpolation gave an expected 86.14/144. In a later development check, Astra and Gemini each scored 30/40. The best tested policy in each router family matched that score. The pair covered 33/40 in hindsight. The routers missed the 32/40 criterion, so the proposed 120-question evaluation did not start.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:LAB-Bench、Haiku、GPT、模型路由,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Ahmed Tariq, Leo Guinan),经 AITOP 多维评分管线结构化提炼。