Cohere 论文：speculative decoding 在细粒度 MoE 模型上收益远超预期
Cohere 实测了 10 个 MoE 模型，发现相邻 token 会大量复用专家，speculative decoding 在低并发 agent 场景下比之前理论预测快得多，做推理服务的人可以看看。
Cohere 的新论文发现，speculative decoding 在细粒度 MoE 模型上即使 batch size 为 1 也有可观收益，此前研究认为此时收益很小。团队测试了 10 个 30B 到 1T 参数的 MoE 模型，并在 H100 GPU 上进行 profile，发现相邻 token 实际共享 20% 到 45% 的专家，远高于独立路由假设下的 4% 到 12%。以 FP8 的 Qwen3-235B 为例，验证步骤的运行速度达到普通解码步骤的 89%，而预测值只有 52%。这意味着 agent 等低 batch 推理场景中，MoE 服务的 speculative decoding 值得实测而非直接排除。
New @cohere paper finds that speculative decoding works well on fine-grained MoE models even at batch size 1, where prior work predicted small gains.
Finds that in mixture-of-experts models, neighboring tokens share 20% to 45% of their experts, against 4% to 12% expected, which makes speculative decoding cheaper than predicted.
Speculative decoding lets a small drafter propose tokens that the big model checks in 1 pass. For MoE models, prior work said checking several tokens loads many extra experts, limiting gains at low batch sizes.
They tested 10 MoE models from 30B to 1T parameters and profiled them on H100 GPUs. Consecutive tokens reuse many experts, and small experts use GPU bandwidth well, so checking extra tokens costs little.
Adjacent tokens shared 20% to 45% of their experts, against 4% to 12% expected from independent routing. For Qwen3-235B in FP8, the verification pass ran at 89% of a normal decoding step's speed, against 52% predicted.
For low-batch MoE serving, such as agent workloads, benchmark speculative decoding on your hardware instead of ruling it out.
