微调 Whisper 帮失语者重建语音：LoRA 让 WER 从 49% 降到 17%
DIY Customized Speech Recognition for a Person with Speech Disabilities
一位脑瘫研究者微调 Whisper 给自己做了实时语音翻译器 AVoz，词错误率从 30% 压到 12%，全程本地运行。代码和训练协议全开源，几小时录音加几天 GPU 时间就能复现，做无障碍应用的开发者别错过。
一篇 arXiv 论文报告了为一位脑瘫合并构音障碍研究者定制的语音识别系统。作者用 LoRA 在消费级硬件上微调 OpenAI 的 Whisper medium 和 large-v3，使用自己录制的语音样本。在 100 条未见过的测试语句上，medium 的词错误率（WER）从 49% 降到 17%，large-v3 从 30% 降到 12%；字符错误率（CER）分别从 26% 降到 7%、从 16% 降到 5%，剩余错误主要是轻微音素替换。
在此基础上，作者开发了 AVoz 移动应用，把他本人的语音实时转成合成语音播放给听者，全部处理在本地完成，音频不出设备。论文还公开了完整的 LoRA 配置、实现代码和逐步录音训练协议，其他言语障碍者或其家属只需几小时标注音频和几天的 GPU 时间即可复现整套流程。
通用 ASR 系统对中重度言语障碍者的识别失败是长期痛点，这项工作用个人化微调给出了可复制的低成本方案，对无障碍技术和开发者社区都有直接参考价值。
DIY Customized Speech Recognition for a Person with Speech Disabilities
General-purpose state-of-the-art automatic speech recognition (ASR) systems fail systematically for speakers with moderate to severe speech disabilities. This paper describes the design, implementation, and evaluation of a customized ASR system built for the first author, a researcher with cerebral palsy and dysarthria. We fine-tuned OpenAI's Whisper medium and large-v3 models using Low-Rank Adaptation (LoRA) on audio samples recorded from the first author's own speech on consumer-grade hardware. On a held-out test set of 100 previously unseen utterances, the corresponding transcriptions resulted in a Word Error Rate drop from 49% to 17% for medium and from 30% to 12% for large-v3, while the Character Error Rate fell from 26% to 7% for medium and from 16% to 5% for large-v3, indicating that the residual errors are minor phonemic substitutions. Based on these models, the first author developed AVoz, a mobile application that acts as a real-time interpreter, converting his speech into synthesized voice for the listener. All processing is performed locally, ensuring that no audio data leaves the user's machine. We argue that the methodology is replicable. Moreover, we publish the full LoRA configuration, the implementation, and a step-by-step recording and training protocol, so that other people with speech disabilities, or their family members and caregivers, can build their own systems with a few hours of labeled audio data and a couple of days of GPU time.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：Whisper、LoRA、语音识别、无障碍，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始信息发布自一手权威信源「arXiv: OpenAI」（Tier-1 一手发布），具备高确定性参考价值。
