CECAR 用缓存感知路由让 MoE 大模型在 16GB 显存显卡上跑出 12.5 tokens/s
CECAR: Cache & Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs
这篇论文有意思的地方是换了思路:不压缩模型,而是预测专家的重用距离来决定路由。Qwen3-30B-A3B 在 16GB 显存的显卡上跑出 12.5 tokens/s,比把模型塞满内存还快。做端侧部署、推理框架的工程师可以直接看开源代码。
arXiv 论文提出 CECAR(Cache & Expert Co-Aware Routing),面向 MoE 大模型的端侧推理。在 Qwen3-30B-A3B 上,CECAR 在仅有 16GB 显存的消费级 GPU 上达到 12.5 tokens/s,非驻留专家权重从 SSD 按需读取,相比传统 MoE 推理加速 4.5 倍,解码速度还超过了把整个模型完整放进 DRAM 的基线方案。
技术上,CECAR 联合考虑专家路由和缓存管理。作者先给出三个观察:即使采用 Belady 最优替换策略,缓存命中率也有上限;该上限不足以支撑低延迟端侧推理;此前基于二值缓存命中与否做路由的方法会导致路由不稳定。CECAR 用机器学习策略预测每个专家的期望重用距离(reuse distance),把路由和执行优先导向重用距离更小的专家,从源头减少昂贵的 SSD 换页。
对想在本地或边缘设备部署 Qwen3-30B 这类 MoE 模型的开发者,这项工作提供了一条不做量化压缩也能跑大模型的路径,代码已在 GitHub 开源。相比单纯扩内存或蒸馏小模型,它把瓶颈从硬件容量转移到调度策略上,对端侧 AI 硬件厂商和推理框架作者都有参考价值。
CECAR: Cache & Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs
Despite growing interest in on-device LLM deployment, large-scale Mixture-of-Experts (MoE) models remain impractical on resource-constrained devices, as sparse expert activation still requires all expert weights to be memory-resident. To address this, we identify three key observations for on-device MoE inference: (a) cache hit rates are fundamentally bounded even under Belady's optimal policy; (b) these bounds are insufficient for low-latency on-device inference; and (c) prior cache-aware routing based on binary cache presence can induce routing instability. Based on these insights, we propose CECAR (Cache & Expert Co-Aware Routing), a MoE inference system that jointly considers expert routing and cache management. Unlike prior cache-aware routing methods, CECAR uses an ML-based cache policy to predict the expected reuse distance of each expert and prioritizes routing and execution toward experts with smaller predicted reuse distances. On the Qwen3-30B-A3B model, CECAR achieves 12.5 tokens/s on a consumer-grade GPU with only 16 GB of VRAM, where non-resident experts are fetched from SSD, delivering a 4.5x speedup over conventional MoE inference and exceeding the decode speed of baselines that keep the entire model fully resident in DRAM. Code is available at https://github.com/hoonki2002/CECAR.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Qwen3、MoE、推理优化、开源模型,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Byeongju Kim, Hoonki Lee, Byungjun Kim, Inyul Ra, Yongjin Kim, Yongseong Lee, Sangyeob Kim),经 AITOP 多维评分管线结构化提炼。