论文多源确认精选

一种无需重新训练的模型知识移除方法

Test-Time Unlearning via Sparse Autoencoder

精选理由

这个方法挺有意思,不用改模型权重就能移除特定知识,在DeepSeek和Gemma上效果还不错,感兴趣的可以看看。

本文提出了一种名为ARIA的新方法,它通过稀疏自编码器在测试时移除大语言模型中的特定知识,而不修改模型权重。该方法在DeepSeek-R1和Gemma-3等模型上进行了测试,在TOFU、R-TOFU和WMDP等基准测试中,成功减少了WMDP-cyber基准的遗忘集准确率,同时保持了MMLU基准的得分仅下降1%以内。

原文 · arXiv: DeepSeek

Test-Time Unlearning via Sparse Autoencoder

Machine unlearning aims to remove specific knowledge from a trained large language model (LLM) without retraining from scratch. Existing methods modify model weights via gradient ascent and its advances. While effective on certain benchmarks, these weight-based approaches exhibit a sharp forget-utility trade-off, where stronger forgetting of target knowledge can degrade model utility, and unlearned knowledge may reappear under post-unlearning fine-tuning or prompt attacks. We propose ARIA (autoencoder-gated inference-time unlearning), a test-time unlearning method that leaves model weights intact and gates access to unwanted knowledge only when generation enters a forget-related state. ARIA uses sparse autoencoder (SAE) latents to train a lightweight linear detector, then applies an interpretable intervention on triggered states with negligible test-time overhead. Empirical evaluations on TOFU, R-TOFU, and WMDP show that ARIA improves the forget-retain trade-off over weight-based baselines across both a thinking model (DeepSeek-R1-Distilled-Qwen-1.5B) and an instruction model (Gemma-3-1B-it), e.g., reducing WMDP-cyber forget-set accuracy significantly while keeping MMLU within 1% of the pre-unlearning model. We further introduce three post-unlearning adversarial attacks targeting weight-space and decoding-space recovery, and find that ARIA remains robust under all three, with forgetting changing by less than 1% under attack. A feature-level case study leveraging the interpretability of ARIA suggests that some retain degradation may reflect response styles underlying the unlearning data rather than leakage of the targeted knowledge itself, highlighting a potential source of bias in unlearning task construction.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Test-Time Unlearning、大语言模型、知识移除、Gemma-3,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始信息发布自一手权威信源「arXiv: DeepSeek」(Tier-1 一手发布),具备高确定性参考价值。截至目前,全网已有 3 家独立信源对该事件进行交叉印证与跟进报道(含 量子位、AI Will 等),属于近期高关注度行业信号。