高UTD比率下价值函数崩溃可恢复
Rank Collapse Is Recoverable, Growing $|Q|$ Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Critic
这篇论文揭示了高UTD比率下价值函数崩溃的可恢复性，提出了基于早期增长的预警方法，能节省约10%的计算资源。
研究人员在Soft Actor-Critic算法中发现，提高更新-数据(UTD)比率会导致两种'塑性损失'：表示崩溃和值函数|Q|增长。
使用宽度为2048且无归一化的缩放批评器，崩溃是可恢复的。在第15k步时，被标记运行的|Q|值比标记水平低一百倍以上，但增长速率已能预测标记(Harrell's C和样本外AUC在Walker2d上为0.
78，在Ant上为0.98)。
LayerNorm批评器降低了这一速率。
Rank Collapse Is Recoverable, Growing $|Q|$ Is Not: Out-of-Sample Early Warning for Value Divergence in High-UTD Soft Actor-Critic
Raising the update-to-data (UTD) ratio breaks off-policy critics in two ways grouped as "plasticity loss": collapsing representations and growing value magnitude $|Q|$. We separate them in Soft Actor-Critic (SAC) with scaled critics (width 2048, no normalization). Collapse is survivable: at UTD ratio 16, HalfCheetah critics with most units dormant keep learning, and the training guard, which stops runs whose loss or $|Q|$ explodes, never flags them. Within one high-UTD SAC configuration, runs start close together, and how far a critic's $\log_{10}|Q|$ has climbed by step 15k, its early growth, ranks the runs by how soon the guard flags them. At 15k, a flagged run's $|Q|$ sits a median of over a hundredfold below its flag level, yet the climb's rate already orders the flags (Harrell's C and out-of-sample AUC 0.78 on Walker2d, 0.98 on Ant, at UTD ratio 4). Dormancy does not. Aborting on this rate saves about a tenth of held-out Walker2d compute and stays net-positive live. A LayerNorm critic lowers the rate, removes the flag on Walker2d at UTD ratio 4 and lowers the return.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：Soft Actor-Critic、UTD、价值函数、强化学习，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.LG」（@Tianqi Bu, YuXuan Peng, Junteng Tu, Henghui Xiao），经 AITOP 多维评分管线结构化提炼。