研究:后训练改变模型内部表征的读出方式而非表征本身
Represented but misread: Post-training changes how language models read out what they represent
这篇论文很有意思:模型内部其实'知道'答案,但后训练把它读歪了。作者用探针加信号检测论量化了这一点,还证明 2000 条标注数据就能扭转读出方向。做模型微调或可解释性研究的人应该看。
arXiv 论文对 6 个开源权重基座模型及其后训练版本进行对比,用线性探针与模型自身回答做信号检测论分析,任务为判断德语文本是人类还是 AI 撰写。探针能很好区分两类文本,但模型回答做不到,甚至常出现反向排序;后训练平均使回答灵敏度下降 Δd' = -0.34。
机制层面,后训练还改变了对给定基率的使用:基座模型对对数几率的偏移小于先验几率的设定变化,多数后训练模型则偏移更大。在两个提供分阶段 checkpoint 的模型中,该变化发生在监督微调阶段,模型拼接定位到早期层,中层之后的变化由文本之后位置的内部状态传递。在可公开复现的否定陈述真值任务上,方向相反:探针在各模型都接近天花板,后训练反而修复了部分基座模型不利用真值信息的回答。
训练研究显示,在微调数据中加入 2,000 条正确标注的新闻文本作者身份判断,能改善报纸和教科书语料上的回答,却在对话语料上使其反向,遵循基座模型新闻探针的方向。结论是后训练改变模型从内部表征读出判断的方式,少量训练信号即可设定其方向。
Represented but misread: Post-training changes how language models read out what they represent
Does post-training change what a language model represents, or how this representation is read out into an answer? We compared linear probes with the models' own answers, analyzed with signal detection theory, in six open-weight base models and their post-trained versions. For judgments of whether German texts were written by a human or an AI, probes and word n-gram classifiers separated the classes well, but the answers did not and often ordered the texts inversely. Post-training lowered the sensitivity of the answers on average ($Δd' = -0.34$ across model families; -0.29 across three prompts). It also changed the use of a stated base rate: Base models shifted their log-odds by less than the stated change in prior odds, most post-trained models by more. In two models with stage-wise checkpoints, the change occurred during supervised fine-tuning, model stitching implicated the early layers, and from mid-depth on internal states at positions after the text, not those of the text, transferred the change between checkpoints. Our corpora cannot be shared, so we add a case reproducible from public data, the truth of negated statements. Here, post-training acted in the opposite direction: Probes decoded truth at or near ceiling in every model, the answers of several base models did not use it, and post-training repaired them; by mid-depth the repair resided at the positions after the statement; below it, the representation of the statement contributed, strongly in one of four models. In a training study, adding 2,000 correctly labeled authorship judgments of news texts to otherwise identical fine-tuning data improved the answers on our newspaper and textbook corpora and inverted them on the chats, where they followed a news-trained probe of the base model. Post-training thus changed how a judgment is read out of what a model represents, and a small training signal can set its direction.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:线性探针、后训练、SFT、可解释性,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Jochen Laubrock),经 AITOP 多维评分管线结构化提炼。