论文提出RC-DS坐标,稀疏自编码器因子恢复率从0.506升至0.913
Readout Scaling Shapes Sparse Feature Recovery
SAE学到什么会被readout缩放改变?RC-DS坐标把因子恢复率从0.506拉到0.913。
arXiv论文提出readout-canonicalized decision-space(RC-DS)坐标,在去除奇异值增益的同时保留决策信息。在受控坐标匹配实验中,当因子在readout缩放之前生成时,条件数100下RC-DS将平均稀疏因子恢复率从0.506提高到0.913,优于同预算的centered-logit训练。在三个固定语言模型checkpoint、18种任务稀疏度设置下,RC-DS的无增益重建在全部设置中占优;centered-logit SAE则在17种设置下更好地重建输出logits。小标签头对照显示,延长训练把重建差距从0.0198缩小到0.0016。一个精确包装器以数值精度复现RC-DS训练,说明提升来自训练几何而非额外信息。
Readout Scaling Shapes Sparse Feature Recovery
Readout scaling changes what a sparse autoencoder (SAE) learns, but the benefit of removing it depends on the data. We use readout-canonicalized decision-space (RC-DS) coordinates to remove singular-value gains while preserving retained decision information. At a fixed readout condition number, changing the generating distribution reverses which training coordinates recover known factors best. In our controlled coordinate-matching study, recovery is highest when training restores the space in which the factors were generated. When factors are generated before readout scaling, RC-DS increases mean sparse-factor recovery from 0.506 to 0.913 at condition number 100, compared with centered-logit training under the same budget. An exact wrapper reproduces RC-DS training to numerical precision, isolating training geometry rather than additional information. In neural policies, RC-DS improves mean logit reconstruction under the tested budgets. On three fixed language-model checkpoints, RC-DS has higher mean gain-free reconstruction in all 18 tested task-sparsity settings, while centered-logit SAEs reconstruct output logits better in 17. Small-label-head controls show that longer training reduces the reconstruction gap from 0.0198 to 0.0016. Together, these results show how readout scaling, the data distribution, and training shape sparse learning. RC-DS provides an information-preserving method for studying and controlling the readout's contribution. Separate label-detection and steering tests distinguish these reconstruction effects from task-specific feature usefulness.