一种新的循环Transformer架构技术报告发布
Looped transformers are a popular architecture topic right now. This new technical report extends t...
朋友，这是Yifan Zhang团队提出的一个新架构，叫Recurrent Looped Transformer (RLT)，它通过让解码器循环处理每个token来提升效率，和传统模型比，在保持每个token计算成本不变的情况下，深度可以随序列增长，挺有意思的。
这篇技术报告提出了一种名为Recurrent Looped Transformer (RLT)的新架构。该架构通过让解码器在每一个token（包括提示和响应）上循环，并使用一个因果编码器构建全局KV内存。对于每个新token，解码器结合其编码器表示、前一个token的最终隐藏状态以及最近激活的滑动窗口缓存。在48层解码器的情况下，计算路径在t个token后通过48t个解码器块，而每个token仍执行固定数量的块。深度随序列增长，但每个token的成本保持不变。该状态转换用于预训练、SFT、采样和RL回放，且在提示-响应边界处不会重置。RL回放使用当前权重重建状态，而不是重用过时的滚动状态。该报告是一个设计提案，作者指出推理增益、硬件加速和RL缩放尚未被测量。
Looped transformers are a popular architecture topic right now. This new technical report extends t...
Looped transformers are a popular architecture topic right now. This new technical report extends the loop across tokens. Recurrent Looped Transformer (RLT) makes the decoder recurrent over every token, prompt and response included. A causal encoder builds global KV memory. For each new token, the decoder combines the token's encoder representation with its own final hidden state from the previous token and a sliding-window cache of recent activations. With a 48-layer decoder, the computation path after t tokens runs through 48t decoder blocks, while each token still executes a fixed number of blocks. Depth grows with the sequence and per-token cost stays the same. The same state transition is used for pretraining, SFT, sampling and RL replay, and nothing resets at the prompt-response boundary. RL replay rebuilds states under the current weights instead of reusing stale rollout states. The report is a design proposal. The author states that reasoning gains, hardware speedups and RL scaling are goals that have not been measured yet. Paper: github.com/yifanzhang-pro… Chat with Paper: academy.dair.ai/papers/recurre… 💬 4 🔄 2 ❤️ 12 👀 1015 📊 6 ⚡
