ICA 认证机制：为 σ 加权后训练加上有效性闸门
Intervention-Certified Amplification: A Validity Gate for $σ$-Weighted Post-Training
这篇 arXiv 论文讲了个挺反直觉的坑：训练时删掉的坏行为会换个形式回来。作者用配对干预把问题量化，再提出 ICA 闸门修好，做后训练的值得看看
论文提出一种 per-sample σ 加权的 SFT/RL 混合后训练方法 DIAA（Dual-Implicit Adversarial Amplification），在锚数据中删除所有 Python fence 后，模型仍会在 MATH-500 的 29% 提示上输出未执行的 Tool-Integrated Reasoning，是 vanilla GRPO 的三倍。配对干预实验显示，抑制 TIR 可恢复 10.6 个百分点的准确率，且基于 0%-TIR 教师锚训练的学生仍会以 45.8% 的比例输出 TIR。为此论文提出 Intervention-Certified Amplification（ICA），用配对 rollout 的目标端证据对 σ 加以闸控，将 TIR 比例降到 13.8% 和 11.2%，并把 MATH-500 准确率从 47.5% 提升到 48.7% 和 49.3%，无需重调即可迁移到 GSM8K。
Intervention-Certified Amplification: A Validity Gate for $σ$-Weighted Post-Training
Hybrid post-training often mixes supervised fine-tuning (SFT) and reinforcement learning (RL), using training-side signals to set each sample's balance. Such signals characterize the anchor data and policy, but not whether the emphasized behavior helps in the target task and evaluation setting. This mismatch can reactivate and amplify pretrained behavior even after its literal form is removed from supervision. We demonstrate this with DIAA (Dual-Implicit Adversarial Amplification), a per-sample $σ$-weighted SFT/RL hybrid. With all literal Python fences removed from its anchor, DIAA emits unexecuted Tool-Integrated Reasoning (TIR) on $29%$ of MATH-500 prompts, three times vanilla GRPO. Paired interventions show $10.6$ percentage points of accuracy recovery when TIR is suppressed and comparable contributions from SFT imitation and $σ$-amplification; a student trained on a $0%$-TIR teacher anchor emits it at $45.8%$. Intervention-Certified Amplification (ICA) gates $σ$ with paired-rollout target evidence. Frozen and online ICA reduce TIR to $13.8%$ and $11.2%$, halve residual drag, improve accuracy from $47.5%$ to $48.7%$ and $49.3%$, and transfer to held-out GSM8K without retuning.
- OpenBMB10-26 13:00原文
