TRIAGE 论文：结合负载定价与语义证据判定的两级 LLM 请求升级机制
TRIAGE: Reliability- and Load-Conditioned Escalation for Two-Tier LLM Serving with a Semantic Evidence Judge
一篇很诚实的服务调度论文：升级成本随负载涨 21.9 倍，加了语义证据判定器后 AUROC 到 0.846，但他们也直接承认预注册假设被证伪、某些模型对不如随机路由。
arXiv 论文 TRIAGE 研究两级 LLM 服务中的请求升级策略，升级决策同时考虑生成后的可靠性和随负载变化的 GPU 价格。在一张 A40 上用 VLLM 运行 Qwen2.5-7B 和 32B（AWQ）的实验中，升级成本从 1 到 10 请求/秒上涨 21.9 倍。预注册的可靠性状态检测损坏证据的假设被证伪，AUROC 仅 0.571，未达到 0.70 的门槛。事后引入的语义证据判定器将检测 AUROC 提升到 0.846（从未服务过的条目上为 0.926），并在 RGB 和三个 RouterBench 模型对上以 0.90-0.96 的 AUROC 预测小模型的错误答案；GPT-3.5 到 GPT-4 的级联在 API 成本降低 79.5% 的情况下与 GPT-4 差距约 3 个百分点，但价格比为 2-3 倍的开源模型对上表现不如随机路由。
TRIAGE: Reliability- and Load-Conditioned Escalation for Two-Tier LLM Serving with a Semantic Evidence Judge
Two-tier LLM serving escalates selected requests from a small model to a large one, but prompt-based routers cannot see whether retrieved evidence supports an answer, and they treat the GPU cost of an escalation as constant. We study TRIAGE, which conditions escalation on a post-generation reliability state and on a load-dependent GPU price, and we report both what held and what did not on one A40 serving Qwen2.5-7B and 32B (AWQ) under VLLM. Escalation cost rose 21.9x from 1 to 10 requests/s but showed no monotone dependence on KV-cache headroom, and a pre-registered claim that the reliability state detects corrupted evidence (AUROC>=0.70) was falsified at 0.571, although any prompt-only router scores exactly 0.500 by construction. Post hoc, a semantic evidence judge raised detection to 0.846 (0.926 on never-served items) and predicted wrong small-model answers at AUROC 0.90-0.96 on RGB and three RouterBench model pairs; the resulting cascade stayed within about 3 pp of GPT-4 at 79.5% lower API cost for GPT-3.5->GPT-4, yet fell below random routing for open-model pairs with 2-3x price ratios. End to end, the 7B->32B pair showed only a +2.88 pp held-out gap, so the allocator rightly declined to escalate and the headline is a null that a small pre-launch gate failed to expose.