RISA:面向块扩散语言模型的可恢复稀疏注意力方法
RISA: Recoverable Sparse Attention for Block-Diffusion Language Models
块扩散模型长上下文慢,是因为每步去噪都要重读整份 KV cache。RISA 让一个块内所有去噪步骤共享一份稀疏读取计划,32K 加速 2.67 倍,128K 到 5.53 倍。做长文本生成、关注扩散语言模型部署的人可以看看,比 MAGE、QUEST、LoSA 同等 KL 下更快。
一篇 arXiv 论文提出 RISA,一种面向块扩散语言模型(DLM)的可恢复稀疏注意力方法,用于缓解长上下文推理中内存受限的注意力瓶颈。
块扩散模型并行精炼多个 token,每个去噪步骤都会重复读取不断增长的历史 KV cache。RISA 利用历史 KV 在块内保持固定、query 随去噪步骤演化的特性,将新生成的历史组织为精确优先层和恢复层,由种子 query 构建受保护的有限读取计划,索引在整个块内复用,注意力则用新 query 和当前块 KV 重新计算。
在 DreamReasoner-8B 上,32K 上下文下默认配置注意力加速 2.67 倍、固定轨迹请求加速 1.
36 倍,128K 下注意力加速达 5.53 倍;
在三个模型的八个设置中,RISA 在相同 KL 下解码速度快于 MAGE、QUEST 和 LoSA,且不改动主干或解码器。
RISA: Recoverable Sparse Attention for Block-Diffusion Language Models
Block-wise diffusion language models (DLMs) refine multiple tokens in parallel, offering an alternative to token-by-token autoregressive decoding. Their long-context inference, however, remains bottlenecked by memory-bound attention: every denoising step repeatedly reads a growing historical key--value (KV) cache. Directly applying sparse attention does not fully solve this problem, because different queries can select different pages, inflating the block's collective read set, while repeated support construction adds overhead and permanent eviction can discard information needed later. We present RISA, a recoverable sparse-attention method with shared read plans for block-wise DLMs. RISA exploits the fact that historical KV remains fixed within a block while queries evolve across denoising steps. Newly generated history is organized into exact priority and recovery tiers; seed queries then build a protected, bounded read plan whose indices are reused across the block while attention is recomputed with fresh queries and current-block KV. This separates history preservation from repeated reading and amortizes sparse-support construction without changing the backbone or decoder. On DreamReasoner-8B, RISA retains comparable task performance. At 32K, the default configuration speeds up attention by $2.67\times$ and fixed-trajectory requests by $1.36\times$ over dense, and attention speedup reaches $5.53\times$ at 128K. Across eight settings on three models, RISA decodes faster than the tested MAGE, QUEST, and LoSA budgets at matched KL.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:RISA、块扩散模型、稀疏注意力、KV cache,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Xinrui Zhong, Fanjiang Ye, Yuke Wang),经 AITOP 多维评分管线结构化提炼。