论文多源确认

DTS:基于差分的分块打分方法加速长上下文稀疏注意力

DTS: Delta-Based Tile Scoring for Efficient Long-Context Sparse Attention

精选理由

处理超长文档或大代码库时,prefill 经常卡在注意力计算上。这篇论文的 DTS 不用训练,打分延迟比 XAttention 低 38%,512K tokens 下还有 3.3 倍加速,精度几乎不掉。做长上下文推理优化的工程师可以看看实现思路。

arXiv 论文提出 Delta-Based Tile Scoring(DTS),一种无需训练、按内容自适应的稀疏分块注意力打分方法,针对长文档和代码库带来的 prefill 瓶颈。方法根据局部向量变化选取 query 和 key 锚点,冗余区域使用更少代表点,异构区域保留更多细节,并通过并行 delta packing 将序列切分为 tile 对齐的分块、并发执行有界局部扫描来加速锚点选择。

正文

在 LLaMA-3.1-8B-Instruct 上的评测显示,DTS 的平均 RULER 和 LongBench 分数与全量注意力的差距不超过 0.10 个百分点,其中 RULER 评测最长到 128K tokens。速度方面,128K tokens 下打分管线延迟比 XAttention 降低 38%;在 4 张 NVIDIA H100 上、512K tokens 场景,相比使用 FlashAttention-2 的 PyTorch 全量注意力基线取得 3.3 倍 prefill 加速。

原文 · arXiv cs.LG

DTS: Delta-Based Tile Scoring for Efficient Long-Context Sparse Attention

Large language models (LLMs) increasingly need to process long documents and codebases, making long-context attention a major prefill bottleneck. Block sparse attention reduces computation by skipping unimportant blocks of the attention matrix, but scoring blocks accurately can itself be expensive. We propose Delta-Based Tile Scoring (DTS), a training-free, content-adaptive tile scoring method for sparse block attention. DTS selects query and key anchors according to local vector changes, using fewer representatives in redundant regions while retaining more detail in heterogeneous regions. To accelerate anchor selection, parallel delta packing partitions the query and key sequences into tile-aligned chunks and performs bounded local scans over these chunks concurrently. The selected tiles are then computed using the original query, key, and value vectors. On LLaMA-3.1-8B-Instruct, DTS keeps average RULER and LongBench scores within 0.10 percentage points of full attention, with RULER evaluated up to 128K tokens. At 128K tokens, DTS reduces measured scoring-pipeline latency by 38% compared with XAttention. On four NVIDIA H100 GPUs, it achieves a 3.3$\times$ prefill speedup over a PyTorch full-attention baseline using FlashAttention-2 at 512K tokens.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:DTS、LLaMA-3.1、FlashAttention-2、稀疏注意力,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Emre Uğur, Jiawen Qi, Qinyu Chen),经 AITOP 多维评分管线结构化提炼。截至目前,全网已有 9 家独立信源对该事件进行交叉印证与跟进报道(含 DAIR.AI、OpenBMB、elvis 等),属于近期高关注度行业信号。