JointServe：在显存受限 GPU 上服务多模型智能体
JointServe: Serving Multi-Model Agents on Memory-Constrained GPUs
一张卡塞不下多个模型就只能换入换出？JointServe 边生成边预取权重、护住 KV，SLO 达标率从 2.8% 拉到 76.8%。
论文提出 JointServe，一种面向多模型智能体的服务系统，把权重分片和 KV 块当作竞争同一份显存预算的驻留对象。系统利用智能体负载的有向调用、间隔执行、双状态驻留三个结构特性，在生成阶段解析调用方输出流中的意图信号来提前预取模型，并用抢占式传输调度器保护活跃会话的 KV。与最强的整体换入换出基线相比，JointServe 将会话完成时间降低 21.7%；在基线全部无法完成的崩溃场景下仍能运行，差距达 46 倍。在重放的 SWE-bench、tau-bench 和 OSWorld 轨迹上，会话完成时间最多缩短 49%，SLO 达标率从 2.8-24.5% 提升到 76.8-100%。
JointServe: Serving Multi-Model Agents on Memory-Constrained GPUs
LLM agents are evolving from a single model calling tools to planners that orchestrate multiple expert models whose capabilities live in distinct weights. This shift breaks a hidden assumption of today's serving systems: that the working set of models fits in GPU memory. On the memory-constrained GPUs typical of private and edge deployments, naive whole-model swapping serializes weight transfer with computation and, without host-backed KV restoration, discards session state at a switch. We present JointServe, a serving system that treats weight shards and KV blocks as residency objects competing for one unified memory budget. JointServe exploits three structural properties of agent workloads (directed calls, gapped execution, and dual-state residency) to prepare models before they are called: an intent signal parsed from the caller's own output stream triggers prefetch during generation, partial k* residency reduces switching to incremental layers, and a preemptive transfer scheduler protects live-session KV with a guaranteed bandwidth share. A carve-out control step adjusts the weight/KV allocation as workload demands change. Evaluations show that JointServe reduces session completion time by 21.7% over the strongest whole-model swap baseline and stays usable in a crash regime where every whole-model swap baseline fails to finish (a 46-fold gap). On replayed SWE-bench, tau-bench, and OSWorld traces, it cuts session completion time by up to 49% and lifts SLO attainment from 2.8-24.5% to 76.8-100%.
