MoE-CORE系统优化内存受限MoE推理
MoE-CORE: Coordinated Expert Offloading and Residency for Memory-Constrained MoE Inference
MoE-CORE系统解决了MoE模型在内存受限设备上的推理问题,性能比vLLM Prefetch提升30倍。
MoE-CORE系统通过协调专家卸载和驻留,解决内存受限的MoE推理问题。在DeepSeek-V4-Flash-W4A8模型上,MoE-CORE的TPOT为38.
0-44.8ms,远低于vLLM Prefetch的1268.
9-1269.1ms。
在GLM-5.2-W4A8C8模型上,MoE-CORE的TPOT为206.
6-220.5ms,优于vLLM Prefetch的5941.
5-5941.8ms。
在84GB NPU内存限制下,最佳配置的TPOT可达21.5ms。
MoE-CORE: Coordinated Expert Offloading and Residency for Memory-Constrained MoE Inference
Sparse expert activation reduces MoE models' computation, yet expert weights can exceed limited device memory. Offloading makes inference feasible on a compact AI appliance but exposes host-to-device transfers to the inference path. We present MoE-CORE, a system that coordinates expert offloading and residency for memory-constrained MoE inference. It stages complete expert layers in alternating buffers during prefill. During decode, it combines nonuniform layer-wise cache capacity, domain-informed initialization, routing-history-aware replacement, and cross-layer prefetching. The main configuration executes router-selected experts exactly; an optional score-based substitution path handles eligible low-score misses. The main comparison uses 1K- and 128-token output caps for MoE-CORE and vLLM Prefetch, respectively. Across five workloads per model, MoE-CORE records a mean time per output token (TPOT) of 38.0-44.8 ms versus 1268.9-1269.1 ms for the evaluated vLLM Prefetch configuration on DeepSeek-V4-Flash-W4A8; the corresponding values on GLM-5.2-W4A8C8 are 206.6-220.5 and 5941.5-5941.8 ms. Under an 84-GB NPU-memory cap, the best measured DeepSeek GSM8K configuration achieves a TPOT of 21.5 ms with approximate expert substitution and multi-token prediction (MTP) at depth 2. These results support coordinated expert residency and transfer scheduling under a device-memory constraint. The code is here.
本条情报归属 AITOP「论文」核心栏目,重点聚焦中国 AI 产业与国产大模型生态。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:MoE-CORE、DeepSeek-V4-Flash-W4A8、GLM-5.2-W4A8C8、推理模型,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始信息发布自一手权威信源「arXiv: DeepSeek」(Tier-1 一手发布),具备高确定性参考价值。