研究提出 MoE 模型投机解码中专家字节去重与最优草稿长度分析
Speculative Decoding for Mixture-of-Experts Models: Expert Bytes, Kernel Bandwidth and the Best Draft Length
想知道 MoE 模型上投机解码到底值不值得开去重？这篇把带宽和最优草稿长度算得明明白白，还用 Lean 验证了推导。
一篇 arXiv 论文研究 mixture-of-experts 模型上的投机解码，指出相邻 token 常选中相同专家，去重引擎每轮只读一次专家权重，比逐槽位引擎请求更少字节。作者把每步时间建模为核请求字节除以有效带宽，并推导出投机快于普通解码的条件，证明最优草稿长度是接受率的单调阶梯函数，主步骤在 Lean 中验证。在修改后的 llama.cpp CUDA、llama.cpp Metal 和 MLX-LM 上实测，逐槽位核有效带宽更高（r=0.32 到 0.75），去重只有在每轮 5 到 8 个 token 时才更快，甚至从不更快。在 DeepSeek-V2-Lite 上，仅用单层专家操作拟合带宽，即可在 CUDA 上对全部 22 个接受率预测出误差不超过 1 的最优草稿长度。
Speculative Decoding for Mixture-of-Experts Models: Expert Bytes, Kernel Bandwidth and the Best Draft Length
Speculative decoding checks several drafted tokens in one verification pass. On a mixture-of-experts (MoE) model, consecutive tokens often select the same experts. A deduplicating engine reads each selected expert once per pass, so it requests fewer bytes than a per-slot engine, which reads it once per token. We write a step's time as the bytes each kernel requests over its effective bandwidth, plus fixed time. Under geometric acceptance, we derive from this step time when speculation is faster than plain decoding. The best draft length is a monotone staircase in the acceptance rate. We bound how a bandwidth error moves each result, and we check the main steps in Lean. Deduplication is faster exactly where its share $\bar u(n)$ of the per-slot bytes is below its ratio $r(n)$ of the per-slot bandwidth. On llama.cpp Metal, MLX-LM and llama.cpp CUDA, which we modified to deduplicate, the per-slot kernel has the higher effective bandwidth (r = 0.32 to 0.75). Thus deduplication is faster only from 5 to 8 tokens per pass, or never. We fit the bandwidths on one layer's expert operation, run alone, in under an hour per model. Bytes from routing traces over these bandwidths predict the time that deduplication saves. The measured saving lies inside its error bound at every point on CUDA and MLX-LM and at 36 of 48 points on Metal. The crossover is the first pass size at which deduplication is faster. The prediction places it within one pass size in 10 of 12 configurations of engine, model and depth. We computed these predictions after the measurements. On DeepSeek-V2-Lite, operation times measured before its speculative runs give the best draft length on CUDA within one at all 22 acceptance values.
