研究：Transformer 的 CoT 训练中自发涌现 Heavy-Ball 加速算法
Transformers with CoT Discover Heavy-Ball Acceleration Through Training
这篇理论论文挺有意思：CoT 推理不只是抄老师算法，训练后自己悟出了比梯度下降更快的 Heavy-Ball 方法，说明推理过程真能发现新算法。
一项 arXiv 论文研究了 Transformer 在线性回归的 in-context 权重预测任务上的 CoT 推理。在 Huang et al. (2025) 的梯度下降教师监督基础上，加入基于真实回归权重的轨迹级监督后，模型学会实现 Heavy-Ball 方法，速度超过原教师算法。该加速算法在没有任何 Heavy-Ball 教师监督的情况下涌现。论文给出了 Transformer 推理能在训练中超越模仿、发现新算法的具体理论案例。
Transformers with CoT Discover Heavy-Ball Acceleration Through Training
Modern transformers often solve complex tasks by generating and refining intermediate reasoning steps. To develop a theoretical understanding of chain-of-thought (CoT) reasoning, a notable recent study by Huang et al. (2025) viewed transformers as algorithm executors, and showed that when transformers' CoT steps are supervised by a gradient-descent teacher, they can learn to implement gradient descent for in-context weight prediction in linear regression. In this paper, we study the same in-context weight prediction task but ask whether CoT reasoning can go beyond merely imitating the teacher algorithm. We show that, after initial supervision from a gradient-descent teacher, further trajectory-level supervision using the true regression weights leads the model to implement the Heavy-Ball method, thereby achieving acceleration over the original teacher. Notably, this accelerated algorithm emerges without supervision from any Heavy-Ball teacher. Our results provide a concrete theoretical example in which transformer reasoning can discover new algorithms through training, rather than merely imitate the algorithm used for supervision.