一种通过 L0 正则化混合专家加速密集 LLM 的方法
Accelerating Dense LLMs via L0-regularized Mixture-of-Experts
这个研究很实用，它提供了一种方法来加速大语言模型，而且速度提升很显著，还能保持性能，对开发者应该有用。
这篇论文提出了一种名为 L0-MoE 的方法，使用 L0 正则化来加速密集 LLM 的推理，速度提升近 2.5 倍，同时性能保持竞争力，优于现有加速基线。该方法通过领域感知的数据集筛选和动态批处理来提高训练效率。
Accelerating Dense LLMs via L0-regularized Mixture-of-Experts
Large language models (LLMs) achieve strong performance but suffer from slow and costly inference. Existing acceleration methods often lead to noticeable performance degradation, while Mixture-of-Experts (MoE) models require extensive computational resources. In this paper, we propose L0-MoE, a lightweight MoE approach using L0-regularization to accelerate dense LLMs nearly without performance loss. Our method introduces a cluster confusion matrix for domain-aware dataset curation and applies dynamic batching for efficient training. Experiments show that L0-MoE achieves up to 2.5x speedup over dense models while maintaining competitive performance, outperforming existing LLM acceleration baselines.