SAE 抑制 PII 泄漏评估方法遭质疑：47% 成功率经多指标验证仅 2.8%
Reducing Logprob Without Causal Verification: Diagnosing SAE-Based PII Suppression in LLMs
做了 SAE 抑制 PII 实验的同学注意，47% 成功率实测只有 2.8% 能扛住多指标检验，评估方法得换。
一篇 arXiv 论文指出，当前用对数概率下降来评估 SAE 特征干预抑制 LLM PII 泄漏的做法存在缺陷，这种指标无法区分真正的特征抑制与大幅激活扰动带来的幅值伪影。研究者在 36 个（模型、层、PII 类型）案例中，将选定的 SAE 特征干预与一致性匹配的随机方向对照组用对数概率和排序感知指标对比。结果显示 17 个案例（47%）在对数概率单一指标下看似成功，但只有 1 个案例（2.8%）通过多指标框架，其余暴露出随机方向反超选定特征、特征选择在 PII 类型间失效两种失败模式。论文还用 9,000 次运行的因果中介可行性研究佐证：验证器能有效区分候选，但相关性筛选出的候选很少能通过。
Reducing Logprob Without Causal Verification: Diagnosing SAE-Based PII Suppression in LLMs
SAE-based feature-level interventions to suppress PII leakage in LLMs are primarily evaluated by log-probability reduction. We show that this criterion cannot distinguish feature specific suppression from a magnitude artifact: a direction-agnostic effect of sufficiently large activation perturbations. Across 36 (model, layer, PII type) cases, we compare selected SAE feature interventions with coherencematched random direction controls using logprob and rank-aware metrics. While 17 cases (47%) appear successful under logprob-only evaluation, only 1 case (2.8%) passes our multimetric framework. The rest reveal two failure modes: random directions outperform selected features, and feature selection collapse across PII types. These results indicate that existing evaluations cannot verify whether interventions target causal pathways of PII leakage. A 9,000 run causal mediation feasibility study corroborates this: the verifier separates candidates reliably, but correlation-selected candidates rarely pass it. We therefore propose a new research mission for NLP: causally grounded PII suppression through circuit-level localization and causal mediation analysis.