OpenBMB 推出 LES 推理技术，MiniCPM-V 4.7 手机端跑出 20.51 tokens/s
35B 的 MoE 模型在手机上跑到 20 tokens/s，靠的是专家预取调度，提速 6 倍多，端侧部署的可以看看实现思路。
OpenBMB 公布 MiniCPM-V 4.7 背后的推理技术 Lookahead Expert Scheduling（LES）。该模型总参数 35B、每 token 激活 3B，专家权重存放在闪存中以适配手机内存。LES 通过跨层路由预测预取专家、重排权重存储和分层混合精度量化来协同优化加载与计算。在 iPhone 18 Pro Max 上，完整方案达到 20.51 tokens/s，相比 3.38 tokens/s 的按需加载基线提速 6.07 倍，且不改变模型架构和专家路由，无需额外训练。
📱 A 35B MoE model running on a phone at over 20 tokens/s.
Meet Lookahead Expert Scheduling (LES), the inference technology behind MiniCPM-V 4.7’s efficient on-device performance.
🧠 The challenge: 35B total parameters, 3B activated per token, and limited device memory. Keeping experts in flash storage makes the model fit, but loading their weights can leave computation waiting.
LES co-optimizes expert loading, compute scheduling, and weight access:
⚡ Expert prefetch and compute scheduling: Cross-layer routing prediction prefetches experts the next layer is likely to need. Experts selected by the actual router that are already loaded compute first, while the remaining required experts load in parallel. Prediction only guides prefetch and scheduling: every required expert is still loaded and used. No additional training is needed.
💾 Weight access optimization: Repack expert weights into contiguous storage to reduce scattered reads, and apply precise layer-wise mixed-precision quantization to reduce weight storage and transfer costs while preserving model capability.
We also adapted DSpark speculative sampling to work alongside LES in the on-device inference pipeline.
In our iPhone 18 Pro Max tests, the full system reaches 20.51 tokens/s, up from a 3.38 tokens/s on-demand loading baseline: 6.07× faster. 🔗 https://t.co/kFD3X4RN0P
The original model architecture and expert routing stay intact. Smarter scheduling makes larger models practical on-device.
👉 MiniCPM-V GitHub: https://t.co/msjm2QlrvX 🤗 Model info: https://t.co/W95QKGhymw 💻 Technical Details: https://t.co/6Ul7qXCgjs
#MiniCPM #OnDeviceAI #OpenSource
