对 Whisper 编码器剪枝 6 层，无需标注数据蒸馏恢复精度
Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery
有人把 Whisper 的编码器砍掉 6 层还不用改推理代码，只用无标注语音蒸馏就把 WER 拉回 20.1%，模型直接可以下载用。
论文提出一种对 OpenAI Whisper 编码器做层剪枝的方法：按 leave-one-layer-out 方式计算每层移除后的词错率（WER）变化，删掉影响最小的 6 层，占编码器层数的 18.5%。剪枝后的模型只是更浅的编码器，不需要任何自定义推理代码。再用无标注单语语音数据做蒸馏恢复精度，四语种平均 WER 从零样本的 21.9% 降到 20.1%，基线为 18.2%。代码和剪枝后的 whisper-large-v3-turbo-encoder-pruned 模型已在 GitHub 和 Hugging Face 开源。
Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery
Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups. For instance, the {\tt whisper-large-v3-turbo} variant reduced the decoder from 32 to 4 layers, while Distill-Whisper similarly reduced the decoder to only 2 layers. Although some attention has been put towards reducing the size of the encoder, no approach has seen wide adoption. This could be due to the need for custom inference implementations to take advantage of the compressed model. We present an approach that ranks encoder layers by the leave-one-layer-out change in Word Error Rate (WER). The six layers that cause the least change are removed, corresponding to $18.5\%$ of the encoder stack. The pruned model requires no custom inference code as it is simply a more shallow encoder with fewer layers. We further distill using unlabeled monolingual speech data to recover performance degradation caused by the zero-shot layer pruning. Mean WER across four languages increases to $20.1\%$ after distillation, compared to $21.9\%$ zero-shot, going from a baseline of $18.2\%$. We release all of our code (https://github.com/rasgaard/whisper-encoder-layer-prune) and the pruned model (https://huggingface.co/rasgaard/whisper-large-v3-turbo-encoder-pruned).
