无标签 RLVR 崩溃机制解析：跨任务耦合导致自我强化集体失稳
Collective Dynamics of Self-Reinforcement: How Cross-Task Coupling Drives Policy Collapse in Unsupervised RLVR
为什么 RLVR 训练一多题就崩？这篇论文用可解模型推导出临界题数公式，还实验验证了移除共享方向能防崩溃，理论派别错过。
论文研究无标签强化学习 RLVR 中模型崩溃的成因。RLVR 让语言模型依据自身多数投票获得奖励，初期提升推理能力，最终奖励趋近 1 而准确率持续下降，模型对所有问题给出相同答案。作者提出集体失稳理论：当问题共享参数时，共享扰动按 ∝λ² 的速率增长，与单个问题的答案锐化竞争，推导出临界问题数 N_c≃ln(1/δ)/(λ²G)。模拟证实了该临界转换，并在小 Transformer 的无标签 RL 中验证：移除共享方向可完全阻止崩溃。
Collective Dynamics of Self-Reinforcement: How Cross-Task Coupling Drives Policy Collapse in Unsupervised RLVR
Label-free reinforcement learning with verifiable rewards (RLVR) rewards a language model for agreeing with its own majority vote. It improves reasoning at first but eventually collapses: the self-assessed reward goes to one while accuracy falls, often until the model gives the same answer to every problem. The collapse appears only when training uses many problems. We explain this dataset-size dependence as a collective instability. In a solvable many-problem model in which problems share parameters, a population whose problems prefer different answers amplifies any shared perturbation for as long as the population-averaged slope $\langle F'\rangle$ of the single-problem self-reinforcement drift $F$ is positive. This holds while problems are still uncertain about their answers. Per-problem sharpening, slowed by $1/N$ for $N$ training problems, races against the shared mode, which grows at a rate $\proptoλ^2$ independent of $N$, where $λ$ is the strength of the shared coupling. This yields a critical number of problems $N_c\simeq\ln(1/δ)/(λ^2G)$, where $δ$ seeds the shared mode and $G$ decreases with the initial confidence. Simulations of the model confirm the predictions: a sharp transition, $N_c\proptoλ^{-2}$ up to a logarithmic correction, a single critical amplification $λ^2NG$ across couplings, confidence profiles and training variants, a logarithmic dependence on batch size, and no collapse for confident initial models. A kernel version of the theory identifies a protective coupling (generalization of sharpening) and a marginal-entropy bonus that stabilizes the uniform collective mode. In label-free RL of a small Transformer, removing the shared direction prevents collapse entirely, restoring it causes collapse, and collapse grows with training-set size and base-model entropy.
