Graph Machine 用边对象替代传统状态，在 Qwen3-0.6B 上预训练时，稀疏层仅需少量检索就能保持性能。
Graph Machine (GM) 是一种保持 O(n) 大小状态的架构，通过稀疏动态路由访问状态。研究人员将 Qwen3-0.6B 中 75% 的密集 Transformer 层替换为 GM 稀疏层，在 157 亿 tokens 上从头预训练。每层每 KV 头仅检索 4096 个 token 中的 2 个时，损失仅略微下降；检索 4 个时，最佳模型损失略有改善。
Graph Machine: Towards Better Pretraining via Edges
We introduce the Graph Machine (GM), an architecture that maintains an $O(n)$-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves $O(n)$ complexity in its sparse layers without restricting the potentially accessible state size to $O(1)$. Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing. We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens. With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.