Anthropic研究揭示模型宽度与特征表示关系

低于临界宽度时，损失随活跃特征数量线性增长；高于临界宽度时，损失随活跃特征数量近似二次增长，且随宽度增加而衰减。

非均匀激活概率会延迟相变并降低损失。

Emergent phases of superposition: from partial to full representation

Large language models are thought to represent features by vectors in a hidden space of dimension given by the model's width. Superposition, in which more features are represented than the width by letting representation vectors overlap, is a leading account of how representation vectors are organized. However, how model width and data statistics determine the configuration of representation vectors and the resulting loss when the number of features and the width are large remains less understood. Here we show, in Anthropic's toy model of superposition, that increasing the width drives a continuous phase transition from a partial-representation phase, where only a subset of features receives appreciable representation vectors while the rest vanish, to a full-representation phase, where every feature is represented. Our theory via a partial random projection approximation predicts, and experiments confirm, that the critical width grows linearly with the number of active features up to a logarithmic factor. The loss scaling changes across the transition: below the critical width, the loss grows linearly with the number of active features and depends weakly on the width in a form set by data statistics; above it, the loss grows approximately quadratically with the number of active features and decays inversely with the width. Non-uniform firing probabilities delay the transition and lower the loss, as more frequent features occupy more space. Our results provide an account of how model width and data statistics jointly shape representations and loss, a step toward understanding representation scaling in large models.

本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：Anthropic、superposition、representation scaling、模型宽度，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明：本文原始信息发布自一手权威信源「arXiv: Anthropic」（Tier-1 一手发布），具备高确定性参考价值。截至目前，全网已有 4 家独立信源对该事件进行交叉印证与跟进报道（含 The Rundown AI、AI Will、Simon Willison’s Weblog 等），属于近期高关注度行业信号。