Ostinato：常驻专家实现 CPU-GPU 混合的 MoE 推理引擎
Ostinato: Fast CPU-GPU MoE Inference for Local Agents via Resident Experts in Every Forward Pass
本地跑智能体的人应该看看这篇。llama.cpp 和 KTransformers 在工具调用后总有一段卡顿等待，Ostinato 把常驻专家放 GPU 上，实测等待时间砍掉 22-55%，用 DeepSeek-V4-Flash 加四张 4090D 就能跑。
论文 Ostinato 针对 MoE 模型在消费级 GPU 工作站上的推理瓶颈提出新方案。现有 CPU-GPU 引擎无法兼顾 prefill 和 decode：llama.cpp 的专家缓存为保住 decode 缓存会跳过超过 32 token 的前向传播，智能体每次工具调用后要等 2.1-2.8 秒；默认层放置虽缩短等待，decode 却慢 1.5-1.6 倍。
核心洞察是：短 prefill 虽然触及大量不同专家，但 84% 的专家激活（路由）集中在一组能放进显存的稳定专家上。Ostinato 让这些专家常驻 GPU、每次前向传播都在 GPU 上计算，且不改变模型计算结果。短 prefill 的路由被拆分，每层留给 CPU 的路由专家不足 1 个（每 token 6 个路由专家）；长 prefill 每个冷专家只上传一次；每个 decode 步骤以固定形状 CUDA graph 运行。
在四块 RTX 4090D 上用 DeepSeek-V4-Flash 跑真实的编程和客服智能体会话，Ostinato 在 prefill 和 decode 两阶段都是最快的引擎，相比 llama.cpp 和 KTransformers，工具输出后的中位等待时间减少 22-55%，会话整体完成速度快 1.4-2.5 倍。
Ostinato: Fast CPU-GPU MoE Inference for Local Agents via Resident Experts in Every Forward Pass
Language-model agents increasingly run on workstations with a few consumer GPUs, where large mixture-of-experts (MoE) models keep part of their experts in host memory. With existing CPU-GPU engines, these agents get fast prefill or fast decode, not both: llama.cpp's expert cache skips forward passes of more than 32 tokens to keep the experts of decode cached, so an agent waits 2.1-2.8 s after each tool output, whereas its default layer placement shortens this wait but is 1.5-1.6$\times$ slower in decode. Our key insight is that a short prefill touches too many distinct experts to upload, yet sends 84% of its expert activations (routes) to a stable set of experts that fits in GPU memory. We present Ostinato, a CPU-GPU MoE inference engine that computes these experts on the GPUs in every forward pass without changing what the model computes. Short prefills split their routes, leaving fewer than one of a token's six routed experts per layer to the CPU; long prefills upload each distinct cold expert once; and each decode step runs as one fixed-shape CUDA graph per GPU. On real coding and customer-service agent sessions with DeepSeek-V4-Flash on four RTX 4090D GPUs, Ostinato is the fastest engine in both phases and finishes every session sooner than all six baselines. Against llama.cpp and KTransformers, it cuts the median wait after a tool output by 22-55% and finishes sessions 1.4-2.5$\times$ sooner.
本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：Ostinato、DeepSeek-V4-Flash、MoE、llama.cpp，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始信息发布自一手权威信源「arXiv: DeepSeek」（Tier-1 一手发布），具备高确定性参考价值。