研究显示：用模型自认为错误的数据微调会诱发广泛失配
Knowingly Wrong: Training LLMs Against Their Own Beliefs can Induce Misalignment
训练数据本身无害但模型不信，也会让它整体变歪，数学任务失配从4%涨到60%。做模型更新的人都该看看。
一篇 arXiv 论文发现，用模型本身不信的虚假数据微调，即使内容无害（如数学、琐事、烹饪、体育领域的错误答案），也会导致广泛失配。在数学任务上，随着训练答案错误程度加深，失配回答比例从 4% 升至 60% 以上，而通用能力几乎无损。微调后模型口头上给出错误答案，但其多选题概率仍偏向正确答案，说明失配源于这种内部不一致。在知识编辑这类更现实场景中同样成立，且模型残留的真实信念越强，失配越严重。现有内容过滤和 persona 向量无法区分导致不同程度失配的两种训练表述方式。
Knowingly Wrong: Training LLMs Against Their Own Beliefs can Induce Misalignment
Fine-tuning a large language model (LLM) on harmful data in a narrow domain, such as bad medical advice, can make it broadly misaligned. How realistic this risk is remains unclear, since models are very rarely fine-tuned on explicitly harmful data in practice. In this work, we show that fine-tuning on data the model believes to be false but that has no harmful content, can also induce broad misalignment. Across mathematics, trivia, cooking and sports, the more strongly the model disbelieves the answers it is trained on, the more misaligned it becomes. In mathematics, misaligned answers rise from 4% to over 60% as the training answers become more wrong, with little loss of general capability. After fine-tuning, the model states the wrong answers but its multiple-choice probabilities still prefer the true ones, which suggests that misalignment comes from this internal inconsistency. We then study a more realistic setting, where a single fact is updated, as in knowledge editing or continual learning. The same relationship holds and the more the model still believes the true fact after training on the false one, the more misaligned it becomes. When we frame the same false fact as part of a coherent story, the model comes to believe it more deeply and becomes less misaligned than when we instruct it to assert the false fact. Content filters and persona vectors cannot tell these two framings apart, even though they lead to different levels of misalignment. These results suggest a risk for model updates, where training a model that does not adopt the new fact as true can induce misalignment even when the data looks harmless.
