论文

变换乘法结构保持参数量:Transformer的关联代数层

Change the Product, Keep the Parameters: Associative Algebra Layers for Transformers

精选理由

MIT团队用新乘法结构提升Transformer速度,参数不变但生成更快,适合关注模型效率的研究者。

研究人员提出了一种基于关联代数的Transformer新架构,使用稀疏交互表替代普通矩阵乘法。实验训练了两个约1.

正文

1亿参数的解码器模型,仅在前馈层使用不同乘法方法。在四个提示域测试中,代数模型实现了6.

2-7.8%的端到端生成吞吐量提升,但在三个下游指标上得分较低。

原文 · arXiv cs.LG

Change the Product, Keep the Parameters: Associative Algebra Layers for Transformers

Fast matrix multiplication algorithms keep the product fixed and search for a cheaper way to evaluate it. We instead ask whether a Transformer's learned projections can use a different, cheaper product altogether. Building on an associative-algebra construction that replaces ordinary matrix multiplication with a sparser interaction table over the same weight blocks, we construct a family with quadratic arithmetic in the matrix dimension when the physical block size remains fixed, and derive finite-shape constraints for GPU execution. The construction is provably optimal for its bilinear rank by the Alder--Strassen bound and can be realized as row-typed rectangular projections compatible with causal masking and KV-cached decoding. We provide an empirical test of this approach by training two approximately 110M-parameter decoder-only Transformer LMs from the same recipe and 12.3B-token budget, differing only in their feed-forward layer: one uses ordinary dense matrix multiplication and the other uses the associative-algebra product. Across four prompt domains, the algebraic model achieves a 6.2--7.8\% increase in end-to-end generation throughput, while obtaining lower scores on all three reported downstream metrics. We treat these results as a feasibility and trainability check for the proposed approach at small scale, leaving further investigation to future work.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Transformer、关联代数、矩阵乘法、生成效率,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Ilya Koziev, Ivan Oseledets),经 AITOP 多维评分管线结构化提炼。