Memory Attention：用 token 记忆表替代 value 投影的新注意力机制
Memory Attention
一篇改注意力机制的论文：value 不再做投影，改查 token 记忆表，推理还能省显存，做模型结构的可以看看。
arXiv 论文 2609.28399 提出 Memory Attention（MA），用层级专属的 token 记忆表与上下文 key 组合来生成 attention value，替代传统 value 投影。推理时归一化可折叠进记忆表，value 构建简化为查找加加法。token 索引检索还支持带预取的 CPU offloading，降低 GPU 参数存储开销。在相同训练 token 预算、增加记忆参数的条件下，多种注意力配置下语言建模和下游平均成绩均有所提升。
Memory Attention
Language models typically construct attention values from contextual hidden states, even when some of their content may be reusable across contexts. We investigate whether token-indexed memory can replace the dedicated value projection when complemented by contextual information. We propose Memory Attention (MA), which forms values by combining layer-specific token memory with contextual keys. The memory supplies token-specific representations, while the keys preserve context dependence. At inference, normalization can be folded into the memory tables, reducing value construction to lookup and addition. Token-indexed retrieval also enables CPU offloading with prefetching, reducing GPU parameter storage. Under matched training token budgets and with additional memory parameters, experiments across attention configurations show improved language modeling and average downstream performance.
