EARL 框架提出表示学习的两种涌现压缩模式
Invariance before structure: two modes of emergent compression in representation learning
这篇论文把神经网络表示的不变性拆成两种压缩模式，还给出时间上的先后证据，做表示学习研究的可以看看。
论文提出 Emergent Adaptive Representation Learning（EARL）框架，把学习表示中的不变性区分为两种模式：状态空间压缩与参数空间压缩，并用 Fisher 敏感性和集成层面的信息论度量将其形式化。在一个带训练瓶颈的循环储备池实验中，参数空间压缩在训练前几个 epoch 内上升，且部分独立于任务；状态空间压缩约在 30 个 epoch 后才出现，且只在网络开始学习任务时间结构时出现。结果表明对底层基质的不变性与任务相关的涌现结构在学习的不同阶段分别获得。
Invariance before structure: two modes of emergent compression in representation learning
Learned representations show striking invariance across biological and artificial neural networks, with different substrates and variations driven by drift, heterogeneity and initialisation. To understand this invariance, we propose Emergent Adaptive Representation Learning (EARL), a framework that distinguishes two modes of emergence in learned representations: state-space compression, where a learned representation gains predictive power over its own future beyond what the network's units can predict individually, and parameter-space compression, where networks with different micro-level parameters converge to the same representation. We formalise both as Fisher sensitivities and introduce ensemble-level information-theoretic measures for each. In a recurrent reservoir with a trained bottleneck, parameter-space compression rises within the first few epochs of training and partly independently of the task, whereas state-space compression rises about 30 epochs later and only when the network begins to learn the temporal structure of the task. These results suggest that invariance to the substrate and task-specific emergent structure are acquired in distinct phases of learning.