NoRA：LoRA单行改进提升模型训练效果
// Normalized Low-Rank Adaptation (NoRA) // They propose a one-line change to LoRA that costs nothi...
DAIR团队提出NoRA，只需一行代码就能改进LoRA，训练更稳定效果更好，还不增加计算负担。
NoRA通过一行代码修改LoRA算法，对下投影矩阵进行归一化处理。该方法在预训练、监督微调和强化学习中均表现出优势，实现更快收敛、更优最终性能、更稳定训练和更少灾难性遗忘。NoRA无需增加可训练参数，也不增加推理时计算量，适用范围广泛。
// Normalized Low-Rank Adaptation (NoRA) // They propose a one-line change to LoRA that costs nothi...
// Normalized Low-Rank Adaptation (NoRA) // They propose a one-line change to LoRA that costs nothing and improves convergence, stability and forgetting. LoRA initializes the up-projection to zero, which means early optimization is governed almost entirely by the down-projection. This observation tells you where to regularize. NoRA normalizes the down-projection matrices during training. The authors also show the same normalization applied once at initialization improves standard LoRA without repeating it through training, which is the cheaper of the two options. The benefits hold across pretraining, supervised fine-tuning and reinforcement learning. Faster convergence, better final performance, more stable training, and less catastrophic forgetting. It adds no trainable parameters and no inference-time computation, which is what makes it broadly applicable rather than another specialized LoRA variant. Paper: academy.dair.ai/papers/normali… 💬 4 🔄 2 ❤️ 9 👀 2090 📊 6 ⚡
