扩散语言模型推理能效约为自回归模型两倍
Energy Consumption Analysis of Discrete Diffusion and Autoregressive Language Models
这篇论文实测了 LLaDA 和 Dream 这些扩散模型的每 token 能耗，长序列下比 Llama、Qwen 省一半电，做推理成本优化的可以看看。
arXiv 论文对比了扩散语言模型（LLaDA-8B、Dream-7B）与自回归模型（Llama-3.1-8B、Qwen-2.5-7B）的推理能耗。在 200-600 token 的较长序列上，扩散模型能耗为 2.5-2.7 J/token，约为带 KV-cache 的自回归模型（5.3-5.7 J/token）的一半。KV-cache 优化能让自回归模型在长序列下降低 50-70% 的能耗。论文分析了模型架构、KV-cache 与每 token 焦耳效率之间的权衡。
Energy Consumption Analysis of Discrete Diffusion and Autoregressive Language Models
As Large Language Models (LLMs) continue to scale, their energy consumption during inference has become a critical concern. While Autoregressive (AR) models are the industry standard, Diffusion Language Models (DLLMs) have emerged as a promising alternative capable of parallel token generation. In this paper, we perform a comparative energy benchmark of diffusion models (LLaDA-8B, Dream-7B) against AR counterparts (Llama-3.1-8B, Qwen-2.5-7B). We analyze the trade-offs between model architecture, KV-cache optimization, and Joules-per-token efficiency. Our results show that at longer sequences (200--600 tokens), diffusion models achieve 2.5--2.7 J/token, approximately 2$\times$ more efficient than AR models with KV-cache (5.3--5.7 J/token). KV-cache provides substantial benefits for AR models at longer sequences (50--70\% energy reduction).
