Primal-Dual 学习中的参数共振：慢变约束 MDP 稳定性分析
Parametric Resonance in Primal-Dual Learning for Slowly Non-Stationary Constrained MDPs
训练带约束的强化学习智能体总不稳定？这篇论文找到了根源：两处约束估计不一致引发的参数共振，还给出了抑制办法。
论文在双时间尺度假设下分析约束 MDP 的 primal-dual 方法，区分加性与参数性两条非平稳通道。参数性通道中，dual 更新的刚度调制会抵消 primal 更新的阻尼调制，不产生共振强迫；失稳来自两个更新所用约束估计不一致，在两倍自然频率处产生参数共振。可容忍的失配随强迫带宽的平方根增长，随 dual-to-primal 步长比的平方根下降。在 Lagrangian soft actor-critic 智能体实验中，乘子锁定在次谐波频率上，相干代价 critic 能抑制该锁定，PID 项则通过乘子投影抑制大部分共振。
Parametric Resonance in Primal-Dual Learning for Slowly Non-Stationary Constrained MDPs
Primal-dual methods for constrained Markov decision processes are analyzed under a two-timescale assumption, with non-stationarity entering through two channels. In the additive channel, where the constraint level oscillates, the dual oscillation saturates at a floor set by the moving optimum and is amplified linearly by the timescale ratio. In the parametric channel, where the constraint sensitivity oscillates, a coherent modulation carries no resonant forcing at any depth: the dual update's stiffness modulation cancels the primal update's damping modulation. Instability instead comes from a mismatch between the constraint estimate used by the primal update and the measurement used by the dual update, producing a parametric resonance at twice the pair's natural frequency. The tolerated mismatch grows with the square root of the forcing bandwidth and collapses with the square root of the dual-to-primal step-size ratio. These results hold on the tangent linear-quadratic reduction around an active saddle, for one constraint and for $K$ constraints under a common modulation and an isotropy condition. Driving both updates from the same estimate removes the resonant forcing but introduces a stationary offset equal to the estimator's bias. In a Lagrangian soft actor-critic agent, the multiplier locks onto the sub-harmonic at the timescale ratios its identified linear loop predicts and a coherent cost critic suppresses the lock-in, while a proportional (PID) term, predicted to leave the lock-in in place, suppresses most of it through the projection of the multiplier.