LIRA：基于损失扰动的可解释动态秩分配 LoRA 微调方法
LIRA: A More Interpretable Dynamic Loss-Informed Rank Allocation for Low-Rank Adaptation
一篇改进 LoRA 微调的论文，用二阶损失扰动给每层打分来动态分配秩，还兼容现有 (A,B) 结构，GLUE 上跑赢了基线，做微调的可以看看。
LIRA（Loss-Informed Rank Allocation）是一种动态秩 LoRA 方法，用二阶近似估计各层 LoRA 因子移动带来的局部损失扰动幅度，并据此为每层打分。它周期性运行 DELTA 算法，在预算匹配的条件下产出离散的有效秩，且兼容标准的 (A,B) 适配器结构。相比现有自适应秩方法依赖启发式或一阶信号，LIRA 的每次秩调整都有逐项可解释的依据。在 RoBERTa-base 的 GLUE 基准上按 La-LoRA/LoRA 协议评测，LIRA 超过均匀 LoRA 和自适应秩基线；其 Trajectory EMA 方案以更低的估计成本达到与 Fixed-point Fisher 相当的性能。
LIRA: A More Interpretable Dynamic Loss-Informed Rank Allocation for Low-Rank Adaptation
Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large pretrained models. However, standard LoRA typically assigns a fixed rank profile despite layer- and stage-dependent adaptation needs, while recent adaptive-rank variants often rely on heuristic or first-order signals that provide only an indirect link between rank reallocation and local loss behavior. This leaves the rationale behind each rank reallocation insufficiently interpretable. Motivated by this gap, we propose LIRA (Loss-Informed Rank Allocation), a dynamic-rank LoRA method designed to make online rank reallocation more interpretable. LIRA scores each layer by the estimated magnitude of the local loss perturbation associated with its recent LoRA-factor movements under a second-order approximation, making the basis for each rank change interpretable term by term. By periodically applying DELTA (Displacement-Evaluated Loss-Trajectory Allocation), a displacement-based algorithm for LoRA rank reallocation, LIRA produces discrete effective ranks under matched budgets while remaining compatible with standard (A,B) adapters. On RoBERTa-base GLUE under the La-LoRA/LoRA protocol, LIRA outperforms the uniform-LoRA and adaptive-rank baselines reported under this protocol; Trajectory EMA attains performance competitive with Fixed-point Fisher at lower estimation cost.
