研究显示 LoRA 模型合并无需统一秩，按域选秩最多提升 1.8 个百分点
Model Merging Does Not Require Uniform LoRA Ranks
arXiv 这篇论文做了 36 个专家、6 种合并方法的大规模实验，证明不同领域的 LoRA 可以用不同秩来合并，做模型合并的值得看看.
论文作者在数学、指令遵循、多语言三个领域训练了 36 个 LoRA 专家，覆盖 4 种秩和 3 种模型规模，再用 6 种方法进行合并测试。结果表明 averaging 和 utility-based selection 两类方法允许每个专家使用各自最优秩，多域准确率最多可提升 1.8 个百分点。而 TIES 这类按幅度选择分量的冲突消解方法则表现不稳定，最多会损失 4 个百分点，其中指令适配器因奇异值最小几乎被整体丢弃，IFEval 成绩下降 16 到 25 个百分点。若改用按对损失的影响来选择分量，准确率在 8 倍秩范围内几乎不变（0.7 个百分点，对比 TIES 在 4B 规模下的 6.7 个百分点）。
Model Merging Does Not Require Uniform LoRA Ranks
Model merging combines separately fine-tuned experts into a single model without retraining. Standard practice trains every expert at the same LoRA rank, which couples the rank choice across experts even though each domain has its own best rank. We show this coupling is a property of the merge method, not of merging. We train 36 experts on three domains (mathematics, instruction following, multilingual) at four ranks and three model scales, then merge with six methods at uniform ranks and at each domain's own best rank. Giving experts different ranks costs averaging and utility-based selection nothing, and gains them up to 1.8 pp of multi-domain accuracy, while the conflict-resolution methods are inconsistent and can lose up to 4 pp. The split has a simple cause. Merging keeps only a fixed share of the pooled spectral components, and methods that choose them by size discard many that would have reduced loss. They also tend to discard the same expert: the instruction adapter has the smallest singular values, so it is dropped almost entirely and IFEval falls by 16 to 25 pp. Choosing components by their effect on loss instead keeps all three experts, and leaves accuracy nearly flat across an 8$\times$ rank range (0.7 pp, against 6.7 pp for TIES at 4B scale). Rank can therefore be chosen per expert whenever the merge operator does not select by magnitude.
