学习长度外推循环模型研究
Learning Length-Extrapolatable Recurrent Models
这篇论文提出了CST方法，能让循环模型在超出训练长度128倍的情况下仍保持良好性能。
研究人员提出了一种名为时间信用稳定化(CST)的新方法，用于解决循环模型在训练范围外性能下降的问题。该方法通过在反向传播过程中局部重新缩放状态信用信号来稳定其范数，同时保持前向计算不变。在受控合成任务和真实数据测试中，CST均表现出色，性能提升可达训练长度的128倍。
Learning Length-Extrapolatable Recurrent Models
Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails. We instead study state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Accordingly, we intervene directly on state credit and propose Credit Stabilization through Time (CST). During backward propagation, CST locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected, while leaving the forward computation unchanged. Because controlled synthetic tasks and real data exhibit different credit dynamics, we specialize CST to each regime. In both settings, CST improves performance beyond the training horizon, with gains observed at up to 128x the training length.
