论文

面向智能体工作负载的 LLM 推理平台验证式部署方法

Validation-driven Deployment of LLM Inference Platforms for Agentic Workloads

精选理由

做自托管推理的团队值得看:论文用数据说明录制回放比实时重跑稳定得多(1.18x vs 4.98x),还实测了 llm-d 上四个 MoE 模型在真实智能体负载下的真实并发和吞吐差距,工具和数据集都开源了。

一篇新论文提出面向智能体 LLM 服务的端到端评估流水线,覆盖从小规模测试、容量规划、预生产到生产的全生命周期验证,目标是帮助企业把智能体工作负载从封闭 API 迁移到自托管的开源权重模型部署。

正文

核心发现是流量回放的价值:回放已捕获的会话轨迹在 95 分位下输入 token 量与原始负载偏差仅 1.18 倍,而实时重跑偏差达 4.98 倍,说明少量录制会话即可产生可复现的单会话 token 负载。对 vLLM 的代码覆盖率分析还显示,实时、回放和合成三类工作负载覆盖的 Python 服务路径互不重叠,各有互补价值。

在一个在 llm-d 上部署四个大型 MoE 模型的案例研究中,捕获的真实智能体工作负载下模型只达到最优并发的一半左右,吞吐量约为公开轨迹测试结果的 60%。相关工具已开源并附带轨迹数据集,可直接用于自托管推理容量的验证。

原文 · arXiv cs.AI

Validation-driven Deployment of LLM Inference Platforms for Agentic Workloads

Enterprises are moving agentic workloads from closed APIs to their own open-weight model deployments. For this reason, lifecycle-wide validation, from small-scale testing through sizing and pre-production to production, becomes all the more important. Typically, rollouts are evaluated with ad hoc tools whose accuracy is rarely checked. We present an end-to-end evaluation pipeline for agentic LLM serving and empirically ground the design choices that connect its stages. Our experiments show that replaying captured traces reproduces the per-session load of the original agent sessions: at the 95th percentile replay stays within 1.18x of the recorded input tokens while live re-runs reach 4.98x, so a small set of recorded sessions yields a repeatable per-session token load. A code-coverage analysis of vLLM further shows that the evaluated live, replayed, and synthetic workloads are complementary in this configuration: each exercises Python serving paths not reached in the measured runs of the others. In a case study rolling out four large mixture-of-experts (MoE) models on llm-d, experiments show that the target models reached about half the optimal concurrency, and correspondingly about 60% of the throughput, on captured workloads compared with public traces. We open-source the relevant tools and release a trace dataset.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:vLLM、llm-d、MoE、推理部署,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Shai Gretz, Alon Halfon, Yoav Katz, Lena Dankin, Michal Jacovi, Rachel Tzoref-Brill, Kfir Toledo, Elron Bandel, Natan Ergas, Michal Malka, Pravein Govindan Kannan, Maroon Ayoub, Michal Shmueli-Scheuer),经 AITOP 多维评分管线结构化提炼。