DETER：预训练阶段主动压制知识，提升模型防篡改能力
DETER: Suppression During Pretraining for Tamper Resistance
做开放权重模型安全的研究者应该看看这篇。它在预训练时插入梯度上升压制敏感知识，1B 模型上对抗性微调的成本比纯语料过滤高 20 倍以上，比事后 unlearning 靠谱得多，代码也开源了。
一项发表于 arXiv 的研究（编号 2611.15479）提出 DETER 方法，核心问题是如何让开放权重模型中可能被滥用的生物、化学、网络安全知识在权重发布后仍然难以恢复。现有做法中，预训练后施加的 unlearning 很容易被微调撤销，而仅靠语料过滤的防护也有限。DETER 的做法是在标准预训练过程中交错插入梯度上升步骤，主动压制目标知识。
实验在 1B 参数规模上验证：经过 DETER 处理后，要让模型在留出的 PubMed 数据上达到未过滤模型的损失水平，所需微调 token 量是仅用语料过滤情况的 20 倍以上。研究同时用留出数据和下游任务评估被保留的知识，并在不同模型规模上测试压制强度，发现微调难度与保留数据损失之间存在 increasingly 有利的权衡曲线。
这项工作把防护关口从预训练后移到了预训练阶段本身，对发布开放权重模型的研究机构有直接参考意义，代码已在 GitHub（MEGlr/DETER）公开。
DETER: Suppression During Pretraining for Tamper Resistance
Open-weight language models can acquire knowledge that could enable biological, chemical, or cyber misuse. Any attempt to limit such knowledge must remain effective after the model weights are released and can be fine-tuned. Unlearning applied after pretraining can be rapidly undone, whereas filtering the pretraining corpus offers greater resistance. We ask whether actively suppressing knowledge during pretraining can further increase the fine-tuning required to acquire it. Even simple gradient ascent interleaved with standard pretraining on the filtered corpus makes medical knowledge harder to acquire through adversarial fine-tuning than filtering alone or post-hoc unlearning. At 1B parameters, reaching the held-out PubMed loss of an unfiltered model requires over 20$\times$ as many PubMed fine-tuning tokens as after filtering alone. Retained knowledge is evaluated using both held-out data and downstream tasks. Across model sizes, varying suppression strength reveals an increasingly favorable trade-off between fine-tuning difficulty and held-out loss on retained pretraining data. These results point to pretraining as an effective stage for making unwanted capabilities harder to acquire through subsequent fine-tuning. Code is available at \href{https://github.com/MEGlr/DETER}{\texttt{github.com/MEGlr/DETER}}.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：DETER、unlearning、开放权重模型、AI安全，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.LG」（@Maria Ios Glarou, Martin Jaggi），经 AITOP 多维评分管线结构化提炼。