Meta 与 MIT 论文：循环复用 Transformer 层可在不增参数下提升模型质量
Meta 和 MIT 这篇论文挺有意思，循环跑同一组层就能变强，adapter 只加 1% 多参数，做小模型优化的话可以看看实验细节。
Meta 和 MIT 的论文验证了循环复用 transformer 层的效果：同一组层反复运行即可增加深度而不增加参数量。团队训练了 1B 参数以内的 dense 模型，训练数据最多 155B tokens，测试了循环位置和次数的影响。实验显示循环整个网络层堆栈在固定大小下质量最好，训练算力受限时循环中间层收益最高。每个循环加入的 adapter 仅增加 1.1% 到 4.3% 参数，相比之下直接加深一倍深度需要增加 51.4% 参数。
New Meta and MIT paper finds reusing a transformer's layers in loops can beat a plain model.
Looping runs the same layers several times, giving a model more depth without more parameters. But each loop costs compute and normally runs in sequence for every token.
They trained dense models up to 1B parameters on up to 155B tokens, testing where to loop and how many times. They also added tiny per-loop adapters and repaired a parallel looping method so loops can run at once during decoding.
Looping the whole stack gave the best quality at a fixed size, while looping middle layers paid off best when training compute was tight. Per-loop adapters added only 1.1 to 4.3% more parameters, versus 51.4% for doubling depth, and helped more as loops increased.
