SAE 对比研究：Mamba 与 Pythia 潜在表征 99.98% 对齐
Comparing Latent Concept Formation in State Space Models and Transformers via Sparse Autoencoders
有人拿 SAE 逐个对比 Mamba 和 Pythia 的内部特征，99.98% 几乎一样，做可解释性的可以看看。
一篇 arXiv 论文用 Sparse Autoencoders（SAE）在 1000 万 token 语料上，对 Mamba-130m 与 Pythia-70m 做特征级对齐分析。在 Jaccard 分布上，99.98% 的 Mamba 特征聚在高对齐边界附近，未发现两种架构存在系统性表征分歧，为 Universality Hypothesis 提供初步支持。Pythia 依靠不受限的注意力能把格式类边缘案例拆成单语义特征，Mamba 则被固定大小的循环状态逼着把无关语法异常压进多语义 junk drawer 神经元。论文判断这 0.02% 的分歧集中在刚性语法解析而非广泛语义本体，架构路由机制对核心语义理解影响甚微。
Comparing Latent Concept Formation in State Space Models and Transformers via Sparse Autoencoders
The quadratic scaling of Transformer self-attention has driven the adoption of sub-quadratic Selective State Space Models (SSMs) like Mamba, which compress past context into a fixed-size recurrent hidden state. This strict informational bottleneck raises a foundational question for mechanistic interpretability: do SSMs and Transformers learn fundamentally distinct latent representations? In this work, we employ Sparse Autoencoders (SAEs) to conduct a large-scale, feature-level correspondence analysis between Mamba-130m and Pythia-70m over a 10-million token corpus. Contrary to hypotheses predicting widespread architectural divergence, we find no evidence of systematic representational divergence between architectures: across the observed Jaccard distribution, 99.98% of Mamba features cluster toward the upper alignment boundary, providing preliminary feature-level support for the Universality Hypothesis. We further identify and qualitatively characterize this microscopic fraction (0.02%) of diverging features, finding patterns consistent with the hypothesis that the recurrent bottleneck selectively limits the parsing of rigid syntax rather than broad semantic ontology. We demonstrate that while Pythia's unconstrained attention permits the monosemantic decomposition of distinct formatting edge-cases, Mamba is forced to compress unrelated syntactical anomalies into polysemantic "junk drawer" neurons to preserve state capacity. Collectively, these results suggest that architectural routing mechanisms may have negligible impact on core semantic understanding, with representational divergence confined to extreme structural margins.
