CiMEO：功能子网络定位降低 LLM 持续学习灾难性遗忘
Functional Subnetwork Localization for Efficient Continual Learning in Large Language Models
一篇论文提出 CiMEO 方法，让 LLM 学新任务时只更新对应的功能子网络，140 万条指令序列训练后遗忘更少，做持续学习可以看看。
论文提出 CiMEO（Function-isolated Memory-Efficient Optimization），一种 LLM 持续学习框架。作者用梯度归因方法定位出互重叠度极低的认知相关功能子网络，训练时将每条样本路由到对应子网络，只更新选定区域的参数和优化器状态。在 140 万条顺序指令、覆盖编程、考试类任务和数学推理的五项生成式基准上，CiMEO 取得对比方法中最高的平均成绩，同时缓解灾难性遗忘。子空间重叠和主角度分析显示，局部化更新有助于保留预训练表示。
Functional Subnetwork Localization for Efficient Continual Learning in Large Language Models
Biological intelligence continually adapts to new information while retaining prior skills through localized plasticity. In contrast, continual learning in large language models (LLMs) remains vulnerable to catastrophic forgetting. A promising direction is to identify functionally specialized subnetworks within LLMs that enable targeted and resource-efficient adaptation. Using gradient-attribution profiling, we localize cognition-associated functional subnetworks exhibiting extremely low mutual overlap. Building on this finding, we propose Function-isolated Memory-Efficient Optimization (CiMEO), a continual learning framework that routes each training example to its corresponding functional subnetwork, restricting parameter updates and optimizer states to selected regions. Across 1.4 million sequential instructions, CiMEO achieves the highest average performance among the compared methods on five generative benchmarks spanning programming, examination-style tasks, and mathematical reasoning, while mitigating catastrophic forgetting. Subspace-overlap and principal-angle analyses further indicate that localized updates help preserve pre-trained representations, highlighting functional subnetwork localization as a practical approach to efficient and stable continual learning in LLMs.
