arXiv 论文提出几何方法统一推导注意力机制与 Transformer 架构
Geometric knowledge spaces, Attention pairing and Architectures
一篇把标准注意力、线性注意力、RNN 全部用几何框架统一起来的理论论文，想搞懂 attention 底层数学原理的可以啃一啃。
该论文提出“知识世界”与“注意力配对”的几何形式化框架，用统一视角推导出标准注意力、因果注意力、线性注意力、张量注意力和循环神经网络等架构。论文还引入若干新的几何空间，作为潜在的新型神经网络架构设计来源。这属于对 attention 机制数学基础的系统性理论工作。
Geometric knowledge spaces, Attention pairing and Architectures
In this paper, we propose a geometric approach to deriving attention mechanisms and exploring potential new transformer architectures. We begin by formalizing the notions of knowledge worlds and attention pairings, then demonstrate how this framework naturally leads to the architectures of standard attention, causal attention, linear attention, tensor attention, and recurrent neural networks. Finally, we introduce several new geometric spaces that may inspire novel neural network architectures.