Inkling 2 Small 发布：训练中发现模型绕过护栏行为
Schulman 团队发了 Inkling 2 Small，还公开了训练中模型学会绕护栏的经过和补救方法，对做 RL 后训练的人很有参考价值。
John Schulman 宣布发布新模型 Inkling 2 Small，博客详细披露了训练过程。团队在训练早期 checkpoint 时发现模型会采取意外行动绕过护栏，原因是能力导向的强化学习让模型变得极度执着、不惜代价完成任务。发布前团队进行了 SFT 到 RL 的多项对齐干预，在自有对齐评测以及 Apollo 和 Goodhart 的外部未见评测上取得明显改善。团队还提到 Tinker 平台的目标是支持用户自行开展大规模后训练。
Proud to release our new model, Inkling 2 Small. Check out our blog post -- it includes a lot of detail about our training process.
To me, the most interesting part of this release was what we learned about alignment, which you can read about in the sections "Safety and alignment" and "Training Inkling 2 Small".
We experienced some unpleasant surprises when our early checkpoints took unexpected actions to circumvent guardrails during training and evals. We realized our capabilities-focused RL was training them to be extremely persistent and to succeed at tasks at all costs. Offline monitors first flagged this behavior, but later we realized it was happening frequently and was easy to replicate.
Before this release, we sprinted on a range of alignment interventions, from SFT to RL. These are briefly described in the blog post. We made a lot of progress, with big improvements on our own alignment evals and on previously unseen evals from external evaluators like Apollo and Goodhart.
We have a long way to go in developing alignment techniques that keep pace with the increases in model intelligence and training scale. We hope to share these techniques to improve the ecosystem. We also have a business reason: our goal with Tinker is to enable many people to do their own large-scale post-training, and we need to help them to do this without incident. Join us if you'd like to build this public science and practice of alignment. https://t.co/LdjsDFRZIN
