SPARE：用路由熵决定 MoE 各层剪枝比例的免训练方法
SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs
MoE 模型太大跑不动？这篇论文的 SPARE 方法不用训练，按每层路由熵决定剪多少专家，Qwen3 剪一半还涨了 6 个点。
SPARE 是一种免训练的 MoE 专家剪枝方法，通过路由熵来决定每层删多少专家，而不是所有层均匀剪枝。论文给出了随机剪枝下的扰动上界，并用固定 selector 的对照实验证明路由越集中输出变化越大。在相同 selector 和剪枝预算下，SPARE 在 Mixtral、Qwen3、DeepSeek 模型上超过均匀分配，高剪枝比例时优势更明显。50% 专家剪枝时，Qwen3-30B-A3B 八任务平均分从 57.2% 升到 63.8%，DeepSeek-R1 的数学、代码、科学平均分从 80.1 升到 82.8。
SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs
Mixture-of-Experts (MoE) architectures scale model capacity efficiently, yet their substantial memory footprint poses a major barrier to deployment. Expert pruning reduces this footprint, but most methods focus on which experts to remove and prune every layer equally. We propose SPARE (Sparsity Allocation via Routing Entropy), a training-free, selector-agnostic method that decides how many experts to remove from each layer using routing entropy, the entropy of tokens' normalized Top-K gate weights on calibration data. When a token's gate weight is spread evenly, removing one active expert discards little of it, so SPARE prunes layers with more diffuse routing more aggressively. A perturbation bound under random pruning formalizes this intuition, and controlled interventions with a fixed selector link concentrated routing to larger output changes. With the same selector and budget, SPARE improves average accuracy over uniform allocation on Mixtral, Qwen3, and DeepSeek models, especially at high pruning ratios. At 50% expert pruning, it raises the eight-task average of Qwen3-30B-A3B from 57.2% to 63.8%, and the math, code, and science average of DeepSeek-R1 from 80.1 to 82.8.