OpenBMB 推出 LES 推理系统，MiniCPM-V 4.7 35B 端侧跑出 20.51 tokens/s
OpenBMB 把 35B 的 MiniCPM-V 4.7 塞进 iPhone，靠 LES 推理系统跑到 20.51 tokens/s，快了 6 倍多，做端侧部署的可以看看实现细节。
OpenBMB 发布 Lookahead Expert Scheduling（LES），用于在手机上运行 MiniCPM-V 4.7 的 35B-A3B MoE 模型。在 iPhone 18 Pro Max 上达到 20.51 tokens/s，比按需加载专家权重快 6.07 倍。方法包括预测预取下一层专家、连续专家布局结合混合精度量化、以及用 DSpark 投机解码一次验证多个候选 token。混合精度量化将权重文件缩小 26.7%，五项基准平均分仅下降 0.16 点。模型架构、专家数量和路由逻辑保持不变。
📱 35B parameters. Real-time inference. On a phone.
Meet Lookahead Expert Scheduling (LES), the inference system powering MiniCPM-V 4.7’s 35B-A3B MoE model on edge devices.
🚀 On an iPhone 18 Pro Max, the full system reaches 20.51 tokens per second, a 6.07× speedup over loading expert weights on demand.
How? Three coordinated optimizations: - 🔮 Scheduling: Predict and prefetch the next layer’s experts while the current layer computes. - 💾 Storage: Combine contiguous expert layouts with mixed-precision quantization to reduce loading overhead. - ⚡ Compute: Use DSpark speculative decoding to verify multiple candidate tokens in one pass.
Predictions guide prefetching; the model’s actual router still determines which experts execute. The architecture, expert counts, and routing remain intact. Mixed-precision quantization reduces the weight file by 26.7%, with just a 0.16-point drop in the five-benchmark average.
LES shows how better inference systems can bring larger, more capable models to everyday devices. 🌍
👉 LES: https://t.co/EVYiHJ7eC5 🔗 GitHub: https://t.co/msjm2QlrvX
#MiniCPM #OnDeviceAI #MoE #EdgeAI
