Meta 公开 Cap-and-Track 推理调度设计,分类服务少用 43% H100
Cap-and-Track: Optimizing LLM Inference Request Dispatch Across Shifting Capacity
Meta 把自家生产环境跑了几个月的推理调度方案写成了论文,亮点是完全不需要预测输出长度,只靠后端拒绝信号自动跟容量。同一流量下省了 43% 的 H100,跑大规模推理服务的团队可以直接抄设计思路。
Meta 在论文《Cap-and-Track: Optimizing LLM Inference Request Dispatch Across Shifting Capacity》中公开了其生产级流量整形系统,核心是让调度器只依靠后端返回的拒绝信号来追踪容量上限,无需预测输出长度或手动重调全局限流。拒绝信号作为饱和资源上的 fail-fast 准入检查,超额请求进入持久优先级队列等待重试;Track 调度器据此向上探测、遇阻退避,自动跟随容量天花板。
生产 A/B 结果显示,同一并发流量下,该系统在 post 和 live-video 理解业务中,用比原来排队固定速率方案少 43% 的 H100 完成了同一分类服务。稳态容量回放测试中,Track 的 goodput 与手动调到超测得容量的固定速率相差不到 1%,但拒绝请求占比只有后者的约四分之一。该系统每天承载超过 3 亿次在线请求和数百 TB 媒体,在补填任务高峰的 24 小时内处理了 11 亿次请求。
论文总结的经验包括:推理引擎内队列只保留刚好填满 batch 的长度、仅凭容量拒绝信号引导调度、保持调度状态、按低优先级任务可等待时长来定机群规模。对运营大规模多租户推理服务的团队,这套机制提供了一条不依赖输出长度预测的容量自适应调度路径。
Cap-and-Track: Optimizing LLM Inference Request Dispatch Across Shifting Capacity
A fixed dispatch rate for LLM inference becomes stale as serving fleets resize and request sizes change. Generative output length is unknown at admission, and shared provider pools can lose capacity. We present Cap-and-Track, a design for asynchronous inference across provider capacity and internal GPUs. Meta's traffic shaper implements Cap-and-Track for post and live-video understanding. Refusals signal the backend's current admission ceiling. Durable priority queues hold excess work outside execution, including refused requests for retry. Cap is a fail-fast admission check at the saturated resource that refuses work before a long wait. Using only those refusals, the Track dispatcher probes upward and backs off. It follows the ceiling without output-length prediction or a fleet-level rate to retune. In a concurrent same-traffic production A/B, the shaper ran per-replica Cap plus Track. It served a classifier on 43% fewer H100s than its queued, fixed-rate precursor, though lower-priority work waited longer at backfill peaks. In steady-capacity real-engine replay, Track ran within 1% of the goodput of a fixed rate tuned above measured capacity, with about a quarter of that rate's refusal share. The shaper carries more than 300 million online requests and hundreds of terabytes of media a day. During a backfill, it handled 1.1 billion requests in its busiest 24 hours. Lessons include keeping queues inside inference engines just long enough to fill batches, steering dispatch by capacity refusals alone, keeping dispatch stateful, and sizing fleets by how long low-priority work can wait.
本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Cap-and-Track、Meta、H100、推理调度,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Benjamin Novotny, Doug Szeto, Chang Wang, Shicheng Chen, George Zhihua Yuan, Bob Kamma, Michael Ge, John Xin Zhou, Andrea Olgiati),经 AITOP 多维评分管线结构化提炼。