论文

arXiv 论文解析:MoE 专家共享何时能改善最终输出

From Local Prediction Error to Final Outputs: A Theory of Expert Sharing in Multi-Step Reasoning

精选理由

做 MoE 的朋友值得看这篇:它给出数学判据说明共享专家省下的局部误差什么时候到最终输出就没了,还用实验复现了局部变好、整体变差的反例。如果你在调专家共享或评估 MoE 可解释性归因,论文里的端到端误差界和干预测试方法都能直接用上。

arXiv 论文《From Local Prediction Error to Final Outputs: A Theory of Expert Sharing in Multi-Step Reasoning》研究了 Mixture-of-Experts 模型中跨层/跨步骤共享专家的问题:共享能降低局部预测误差,但多步系统的评估标准是最终输出,局部收益未必在组合后保留。

正文

论文在指定线性回归设定下推导出精确的共享与合并判据,权衡方差节省与跨角色共享带来的偏差;在固定线性传播映射下刻画了最终输出的期望平方风险,并构造出共享降低平均局部误差却升高最终输出误差的严格风险反转案例。对一般路由计算,论文给出考虑分布偏移、周边组件误差与专家选择变化的条件化端到端误差界。

实验部分显示,低数据符号设定下训练的 Transformer 在 teacher-forced 单步评估中有统计显著的收益,但自由运行时的最终输出收益未被统计分辨。论文还指出仅凭正确预测和路由标签无法证明单个专家实现了被归因的操作,需要干预测试来区分多层动作、单专家归因与对未见转移的泛化。

这项工作对 MoE 架构设计者有直接参考价值:专家共享策略应基于端到端误差界而非局部损失来决策。

原文 · arXiv cs.AI

From Local Prediction Error to Final Outputs: A Theory of Expert Sharing in Multi-Step Reasoning

Mixture-of-experts (MoE) models can share experts across layers or computational steps, pooling training information from different contexts. This can reduce local prediction error, but multi-step systems are ultimately evaluated by their final outputs. We study when these local gains survive composition. In specified linear regression models, we derive exact sharing and merging criteria by balancing variance savings against bias from sharing across roles with different target behavior. Under a common fixed linear propagation map, we characterize expected squared final-output risk, retaining error directions and correlations. We construct cases where sharing lowers average local error yet increases final-output error. Within this setting, we also identify when orthogonal full pooling with isotropic noise preserves every local improvement at the final output. For general routed computations, we derive conditional end-to-end error bounds that account for distribution shifts, errors in surrounding components, and changes in expert selection. Separately, we show that correct predictions and route labels alone do not establish that individual experts implement the attributed operations. Controlled experiments realize a strict local-to-final risk reversal. In low-data symbolic settings, trained Transformers exhibit statistically supported teacher-forced one-step gains without statistically resolved free-running final-output gains. Separate intervention tests distinguish successful multi-layer actions from single-expert attribution and generalization to unseen transitions. Together, these results distinguish when sharing improves local estimation, when those gains survive to the final output, and what evidence is needed to attribute reusable operations to individual experts.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Mixture-of-Experts、MoE、多步推理、Transformer,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Tiantong Wang, Zhiren Gong, Yixin Wang, Tiantong Wu, Wei Yang Bryan Lim),经 AITOP 多维评分管线结构化提炼。