Graph Metanetworks 将权重空间学习扩展到 Transformer 语言模型
Graph Metanetworks for Transformer Based Language Models
一篇把权重当数据来学的论文，把 MLP 上成熟的做法补齐到了 decoder-only 语言模型，还顺手放了两个新数据集，做 weight-space 研究的可以看看。
这篇 arXiv 论文针对 Metanetworks 处理 Transformer 权重的问题提出新方法。作者形式化了 decoder-only 语言模型的完整权重空间对称群，覆盖注意力矩阵的连续 general-linear 对称性和注意力头的置换。基于 graph metanetworks 构建了对该对称群不变的图结构，可跨不同深度和宽度使用。团队还发布了两个新的 transformer-population 数据集，包含训练语料预测和微调检测等新任务，方法在这些任务上明显超过现有基线。
Graph Metanetworks for Transformer Based Language Models
Weight-space learning treats the weights of neural networks as data, processed by other models called metanetworks. A key challenge is that different weights can realize the same function, related by a group of symmetries the metanetwork should respect. While symmetry-aware metanetworks for simple architectures such as MLPs are well developed, a comprehensive treatment of transformers, and in particular transformer-based language models, is still missing. Importantly, their weight space has a richer symmetry group: beyond neuron permutations, attention introduces continuous general-linear symmetries of the attention matrices and permutations of entire heads. Existing methods for transformer weights either capture only part of these symmetries or are restricted to a single, fixed architecture, and none accounts for language-model-specific components such as the token embeddings. Building on graph metanetworks, we propose a graph-based approach that is both symmetry-aware and applicable across depths and widths. We formalize a weight-space symmetry group of complete decoder-only language models and provide a graph construction that is invariant to it. We further introduce two new transformer-population datasets with novel tasks, such as training-corpus prediction and fine-tuning detection, on which our approach notably outperforms existing baselines.
