NVIDIA 提出自适应 Megakernel 方案,Nemotron 3.5 Transcribe 推理吞吐最高达 6367 倍实时
High-Throughput Serving of Nemotron 3.5 Transcribe with Regime-Aware Megakernels and Kernel Fusion
NVIDIA 给自家 Nemotron 3.5 Transcribe 换了个自适应推理引擎,每个 pass 自动在 megakernel 和融合 kernel 之间切换,B200 上跑到 6367 倍实时,比 vLLM 快 64%。做语音转写或推理优化的人都该看看这套 per-pass 切换的思路。
NVIDIA 发布面向 Nemotron 3.5 Transcribe(Hugging Face Open ASR 开源模型榜第一)的高吞吐推理引擎。在 H100 SXM GPU 上,精度无损的推理吞吐峰值达 4,345 倍实时,为调优后 vLLM 峰值的 1.39 倍;B200 上峰值达 6,367 倍实时,在最高并发下为 vLLM 的 1.64 倍,WER 保持在 1.91-1.93%,与 vLLM 的 1.88-1.93% 基本持平。
技术上,连续批处理会在每次迭代改变工作负载的规模与构成,单一融合粒度无法一直最优。该引擎针对每个 pass 在离线测得的交叉点处选择持久化 megakernel 或融合 kernel 链:128 并发时比始终用 megakernel 高 23% 吞吐,8 并发时比始终用融合 kernel 高 9%;在 400 ms p95 端到端时延预算内可服务的并发请求数是 vLLM 的 8 倍。B200 上还可开启贪心 MoE 层跳过,对离线任务额外提升 13-16% 吞吐,代价是 WER 略升。
对部署语音转写服务的企业与推理工程师,这套 per-pass 切换思路可直接迁移到其他生成式模型的 serving 优化。
High-Throughput Serving of Nemotron 3.5 Transcribe with Regime-Aware Megakernels and Kernel Fusion
Nemotron 3.5 Transcribe ranks first among open-weight models on the Hugging Face Open ASR leaderboards. Serving it efficiently takes more than fast operators: continuous batching changes the amount and mix of work at every iteration, and the best fusion granularity changes with it. We present a serving engine that, for every pass, chooses between persistent megakernels and chains of fused kernels at crossovers measured offline. On an H100 SXM GPU, throughput of our accuracy-preserving serving peaks at 4,345x real time, 1.39x the peak of tuned vLLM, and is up to 4.39x faster at low concurrency; on B200 it peaks at 6,367x, 1.64x vLLM at the highest concurrency we measured. Word error rate (WER) is 1.91 to 1.93% against 1.88 to 1.93% for vLLM. Within a 400 ms budget for p95 end-to-end completion time, it serves 8x as many concurrent requests. Choosing per pass beats either level alone: 23% higher throughput than always using megakernels at 128 concurrent requests, and 9% higher than always using fused kernels at 8. Opt-in switches, including greedy MoE layer skipping, add 13 to 16% throughput on B200 for offline jobs that can accept higher WER.
本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Nemotron、vLLM、NVIDIA、语音识别,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Aditya Chatterjee, Sanjay Singh Chauhan, Mayank Jain, Arushi Dhingra, Sahin Chowdhury, Rahul Mittal),经 AITOP 多维评分管线结构化提炼。