# Six Layers Less：Whisper 编码器剪枝与无标签恢复方法

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-09-23 08:00
- AIHOT 分数：36
- AIHOT 链接：https://aihot.news/items/cmufbne990520roagttqthty2
- 原文链接：https://arxiv.org/abs/2609.27980

## AI 摘要

研究者提出按"留一法"对 WER 影响排序来剪除 Whisper 编码器层，去掉影响最小的 6 层，占编码器堆栈的 18.5%，剪枝后模型无需自定义推理代码。再用无标签单语语音数据蒸馏恢复性能，四种语言平均 WER 从零样本剪枝的 21.9% 降至 20.1%，基线为 18.2%。代码与剪枝模型已开源。

## 正文

Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups. For instance, the {\tt whisper-large-v3-turbo} variant reduced the decoder from 32 to 4 layers, while Distill-Whisper similarly reduced the decoder to only 2 layers. Although some attention has been put towards reducing the size of the encoder, no approach has seen wide adoption. This could be due to the need for custom inference implementations to take advantage of the compressed model. We present an approach that ranks encoder layers by the leave-one-layer-out change in Word Error Rate (WER). The six layers that cause the least change are removed, corresponding to 18.5% of the encoder stack. The pruned model requires no custom inference code as it is simply a more shallow encoder with fewer layers. We further distill using unlabeled monolingual speech data to recover performance degradation caused by the zero-shot layer pruning. Mean WER across four languages increases to 20.1% after distillation, compared to 21.9% zero-shot, going from a baseline of 18.2%. We release all of our code (https://github.com/rasgaard/whisper-encoder-layer-prune) and the pruned model (https://huggingface.co/rasgaard/whisper-large-v3-turbo-encoder-pruned).