NVIDIA 发布 PyTorch MoE 训练优化指南，用分组执行减少内核启动开销
NVIDIA 出的 MoE 训练优化教程：HF 基线实现循环启动内核很慢，改成分组执行能省不少开销，跑 MoE 的可以看看。
NVIDIA 发布博客，讲解如何在 PyTorch 中对 MoE 模型做性能优化。文中指出 Hugging Face 的基线实现用 Python 循环遍历所有专家，每个专家触发单独的内核启动，开销过大。分组执行（Grouped execution）保留各专家矩阵，但把计算合并提交，降低启动与调度开销。博客以生物基础模型为例演示高效的 MoE 训练。
PyTorch operations can be performance optimized for MoE models applications.
MoE models replace a single dense feed-forward block with multiple expert networks. A naive implementation, however, can trigger excessive kernel launches. For example, the Hugging Face baseline implementation iterates over all experts in a Python loop, with each expert triggering separate kernel launches.
Grouped execution preserves the individual expert matrices but submits their work together, reducing launch and scheduling overhead. This blog from @nvidia uses biological foundation models as an example for efficient MoE training.
Read the full post: https://t.co/IwubEgaFZS
