新研究：模型越大，量化剪枝造成的损伤越难被网络自行补偿
When Bigger Becomes Brittle: Why Compression Hurts More as Language Models Scale
研究了量化剪枝为什么在大模型上更伤——用 attention heads 消融分解把补偿和真实损失拆开了，做模型压缩的可以看看。
一篇 arXiv 论文研究量化与剪枝对语言模型内部的影响。作者聚焦 in-context retrieval 这一能力，将其定位到一组可识别的 attention heads 上，并对消融这些 heads 造成的期望损失做精确分解，区分网络其余部分补偿的部分和最终到达输出的部分。跨模型规模比较发现：损伤规模相近时，小模型能恢复大部分损失，而大模型保留大部分损失，因为补偿性响应的抵消作用随模型变大而减弱。对 heads 压缩得到同样规律，说明在小模型上近乎无害的压缩在大模型上可能代价高昂。
When Bigger Becomes Brittle: Why Compression Hurts More as Language Models Scale
Quantization and pruning reduce the cost of deploying language models, but their effects are usually reported as benchmark scores. Such scores record only the loss that remains at the output. Inside the model, compression damages some components more than others, and the rest of the network can partly compensate. The observed loss therefore reflects both the damage and this compensation, which benchmark scores cannot separate. How the two combine, and whether their balance changes with model size, remains unclear. We study this compensation for in-context retrieval, a capability carried by a small, identifiable set of attention heads. To separate the two, the expected loss from ablating these heads is decomposed exactly into the part restored by the rest of the network, attributed to the components responsible, and the part that reaches the output. This decomposition is then compared across model sizes. For damage of comparable size, smaller models restore most of the loss, whereas larger models retain most of it, because the compensating responses offset less of it as models grow. Compressing the heads produces the same pattern. Compression that is nearly harmless in small models can therefore be costly in large ones.
