开源模型注意力架构加速转向混合与稀疏设计
Raschka 统计了开源模型注意力架构的演变：现在已没有纯全注意力的新模型了，一半完全换掉，一半只留少量全局层。做长上下文、代码库或 Agent 应用的开发者值得看看这两类架构在显存和检索精度上的取舍。
Sebastian Raschka 在整理开源 LLM 年度盘点时指出，2026 年下半年新发布的开源模型中，使用完整全局注意力的模型已降为零，"保留部分全注意力"的混合架构与"完全不用全注意力"的架构各占 50%。相比 2025 年仍有大量全注意力模型、2026 上半年混合架构逐步增多，这一转变在半年内明显加速。
按其分类，"Some full attention" 指全局注意力与滑窗/分块注意力、Gated DeltaNet、KDA、Mamba-2、Lightning Attention、卷积或稀疏注意力混合使用；"No full attention" 则完全不保留稠密全局层。效率收益来自两方面：稀疏注意力只对选中的 token 子集执行昂贵的注意力计算，仍能检索远距离信息；Gated DeltaNet 等循环层维持固定大小的状态，显著降低上下文显存，但压缩历史会影响精确检索，因此通常与传统注意力混合。
这一趋势的核心动因是降低长上下文使用成本，使架构更适合长文档、长对话、大型代码库，以及依赖大量工具调用的智能体场景。
Currently working towards a bigger open LLM roundup. But in the meantime, just one of the interesting trends that stood out. I.e., looks like the transition from full to hybrid attention continues.
In 2025, there were still lots of full attention models, with a shift to "some full attention" hybrids.
In the first half of 2026, the trend towards "some full attention" hybrids continued.
And now, in the second half of 2026, we have a 50/50 split between "some full attention" hybrids and models that don't use full attention at all anymore. And yes, there are also no full attention models anymore.
The plot legend here means,
- Full attention throughout = Global MHA, GQA, dense MLA, CCA in EVERY layer. - Some full attention = Full global attn mixed with: sliding-window/chunked attention, Gated DeltaNet, KDA, Mamba-2, Lightning Attention, convolutions, sparse attention. - No full attention = Zero dense global layers.
The main implication is that the trend goes towards making long contexts cheaper to use. This means, it makes the architectures more suitable for long documents and conversations, large code bases, and of course agent usecases that rely on a lot of tool calls.
In particular, there are two distinct efficiency gains here:
1) Less attention computation, since sparse attention and the like select tokens or blocks from the context and performs the expensive attention operation from only that subset. (Note that it can still retrieve information from distant positions.)
2) Less memory for retaining context in some architectures, meaning that recurrent layers such as Gated DeltaNet maintain a fixed-size state. Of course, this saves memory, but compressing the history creates challenges for exact retrieval, so they are usually intermixed with traditional attention methods (sparse or full).
本条情报归属 AITOP「行业」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：注意力机制、Mamba-2、Gated DeltaNet、开源模型，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「Sebastian Raschka」（@rasbt），经 AITOP 多维评分管线结构化提炼。