论文

GEAR²:让被剪枝的 MoE 专家通过门控复用继续发挥作用

GEAR$^2$: Pruned Experts Live On through Gated Reuse of Activated Experts and a Missing-Mass Residual

精选理由

剪 MoE 专家不用再花钱做恢复训练了。GEAR 直接复用已激活专家的输出,GEAR² 加个轻量残差补缺口,在 16B 到 106B 的五个模型上验证有效,剪一半专家性能还能保住。做模型部署和显存压缩的工程团队可以试试。

一篇 arXiv 论文提出 GEAR 和 GEAR² 两种免训练方法,用于在专家剪枝后恢复被丢弃的 MoE 专家贡献。

正文

作者发现被剪枝专家的功能可以从剩余激活的专家中重建,GEAR 通过门控直接复用可靠激活专家的缩放输出,无需训练或额外专家计算。GEAR² 在此基础上加入一个超轻量残差,补偿复用无法覆盖的部分。

在 16B 到 106B 参数的五个 MoE LLM 上剪枝 25% 和 50% 专家时,GEAR 在九个零样本基准上准确率比免训练基线最高提升 4 个百分点,GEAR² 在所有设置下优于基线方法最高 2.75 个百分点,剪枝越激进(最高 50%)优势越明显(最高 2.

72 个百分点)。相比静态替代或专家合并方案,该方法不改变专家原有功能,也省去了昂贵的恢复训练,对需要压缩 MoE 模型显存占用的部署方有直接参考价值。

原文 · arXiv cs.AI

GEAR$^2$: Pruned Experts Live On through Gated Reuse of Activated Experts and a Missing-Mass Residual

Mixture-of-Experts (MoE) Large Language Models (LLMs) activate only a few experts per token while generally keeping all the experts in GPU memory for high efficiency. Expert pruning reduces this GPU memory footprint cost, yet discards the contribution of pruned experts. Existing remedies either rely on static substitutes, merge experts at the cost of altering their functions, or require costly recovery training. In this paper, we find that the functionalities of pruned experts can be efficiently reconstructed from the activated remaining experts. We propose GEAR (Gated Expert Activation Reuse), which reuses the scaled outputs of reliable activated experts without any training or additional expert computation. In addition, we propose GEAR$^2$ to further recover what reuse cannot express with an ultra-lightweight residual. On five MoE LLMs from 16B to 106B parameters with 25% and 50% of experts pruned, GEAR improves over its training-free counterparts in nearly every setting (up to 4 pp.) in terms of accuracy, and GEAR$^2$ outperforms baseline approaches (up to 2.75 pp.) in every setting on nine zero-shot benchmarks, with larger advantages (up to 2.72 pp. higher) under more aggressive pruning (up to 50%).

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:MoE、模型剪枝、LLM 推理、零样本评测,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Linliang Chen, Yan Zhong, Wang Kang, Sai Li, Ji Wu, Ji Liu),经 AITOP 多维评分管线结构化提炼。