论文

CoPre-MoE:面向多模态持续指令微调的专家合并与保留方法

CoPre-MoE: Consolidating and Preserving MoE for Multimodal Continual Instruction Tuning

精选理由

多模态模型连续学新任务,传统 MoE 会专家越堆越多。这篇把重复专家合并、把任务专家拆分压缩,还用冻结检索键防遗忘。做持续学习或多模态微调的工程师和研究者都该看看这套共享-私有专家池设计。

论文提出 CoPre-MoE,解决多模态大语言模型持续指令微调中的灾难性遗忘与参数冗余问题。现有 MoE 方法为每个任务单独分配专家,虽避免覆盖共享参数,但导致专家数量随任务线性增长,且 ScienceQA 和 TextVQA 等任务间常出现功能重叠。

正文

CoPre-MoE 的核心机制分三步:先为每个新任务学习任务专属 MoE;再通过功能重叠评估,将跨任务重复的专家合并进固定容量的通用专家池;最后把剩余任务专家分解为共享投影基和紧凑的任务核心,放入私有专家池。

为缓解遗忘,路由器为每个专家配可学习检索键,历史键在持续更新中保持冻结,新增键用正交性惩罚约束。推理时基于输入动态组合两个专家池,无需测试时提供任务标签。实验显示该方法在降低参数开销的同时取得更优性能。

原文 · arXiv cs.LG

CoPre-MoE: Consolidating and Preserving MoE for Multimodal Continual Instruction Tuning

Multimodal continual instruction tuning requires multimodal large language models to sequentially acquire new capabilities without catastrophic forgetting. Mixture-of-experts (MoE) architectures offer a promising way to mitigate task interference through expert specialization and combination. However, while existing MoE-based methods avoid overwriting shared parameters by allocating separate experts for each task, this progressive expansion incurs severe cross-task redundancy and linear parameter growth. Specifically, experts across tasks often learn overlapping functions alongside task-specific capabilities. For instance, ScienceQA and TextVQA both locate relevant image regions. To curb parameter redundancy without inducing forgetting, we propose Consolidating and Preserving MoE (CoPre-MoE). For each incoming task, CoPre-MoE first learns task-specific MoE to capture specialized capabilities. To eliminate redundancy and reduce parameter growth across tasks, we design a shared-and-private expert architecture. Specifically, we first evaluate functional overlap to selectively consolidate overlapping experts across tasks into a fixed-capacity generic expert pool. We then factorize the remaining task-specific experts into shared projection bases and compact task-specific cores within a private expert pool. To mitigate forgetting, we construct the router by pairing each expert with learnable retrieval keys, where historical keys remain frozen during continual updates while newly added keys are regularized with an orthogonality penalty. At inference, input-dependent routing dynamically combines both pools without test-time task labels. Extensive experiments demonstrate that CoPre-MoE achieves superior performance while substantially reducing parameter overhead.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:CoPre-MoE、MoE、多模态、持续学习,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Tao Hu, Ling-Xiang Zhang, Da-Wei Zhou),经 AITOP 多维评分管线结构化提炼。