arXiv 论文提出 opinion leader dynamics，解释稀疏注意力如何塑造 token 聚类
Opinion Leader Dynamics: How Sparse Attention Shapes Token Clustering
把 token 当粒子建模，还拿 Kimi-K3 和 DeepSeek-V4-Flash 实测验证，讲清稀疏注意力为啥省算力还不丢效果。
一篇 arXiv 论文把 token 建为单位球面上的粒子，提出 opinion leader dynamics 框架，从理论上解释稀疏注意力为何能让 token 组内收敛、组间保持分离。作者把显式与隐式两类模型表述为 reverse Wasserstein gradient flow，并在适当条件下证明指数收敛。实验对比了稀疏注意力的 Kimi-K3、MiniMax-M3、DeepSeek-V4-Flash 与稠密注意力的 GLM-4.7-Flash，在四个基准上稀疏模型投影后的 token 呈现更清晰的簇分离和更高聚类分数。论文据此给出动力学解释：受限的 token 交互对应组级吸引子，使稀疏注意力能同时支持组内对齐与组间区分。
Opinion Leader Dynamics: How Sparse Attention Shapes Token Clustering
Sparse attention reduces the quadratic cost of global self-attention while retaining strong empirical performance, but how its restricted interactions shape the evolution of token representations remains theoretically underexplored. Modeling tokens as particles on the unit sphere, we introduce opinion leader dynamics, a framework that identifies two mechanisms through which token groups converge internally while maintaining distinct limiting directions. In the explicit model, fixed representatives induce a potential that attracts tokens toward distinct local maxima. In the implicit model, disconnected interaction groups evolve toward separate consensus directions. We formulate both models as reverse Wasserstein gradient flows and establish exponential convergence under suitable conditions. We further connect these theoretical predictions to token evolution in frontier sparse-attention LLMs that motivate our framework. Across four benchmarks, Kimi-K3, MiniMax-M3, and DeepSeek-V4-Flash consistently exhibit clearer cluster separation and higher clustering scores than the dense-attention model GLM-4.7-Flash in projected token representations. These observations support the relevance of the predicted multiple-group structure to trained frontier LLMs, while finite-particle simulations illustrate the theoretical convergence behavior. Together, our results connect restricted token interactions to distinct group-level attractors, providing a dynamical account of how sparse attention can support alignment within groups while preserving separation between them.
