约束部分可观测强化学习首获非渐近有限时间保证
Finite-Time Guarantees for Constrained Reinforcement Learning under Partial Observability
做安全强化学习或 POMDP 的朋友看看这篇：用有限历史窗口近似 CPOMDP，首次拿到奖励和约束违规同时有界的非渐近保证，理论味道很浓。
论文研究带成本约束的部分可观测马尔可夫决策过程（CPOMDP），其中智能体只能基于不完整的状态信息最大化奖励并满足约束。作者利用 POMDP 的有限记忆近似方法，把有限观测-动作历史窗口建模为约束 MDP 的状态，并设计出原始-对偶算法。该算法首次给出同时约束原始 CPOMDP 中平均奖励次优性与约束违反的非渐近保证。在滤波稳定性假设下，近似误差随窗口长度按几何速率衰减，部分可观测网格世界实验验证了方法的有效性。
Finite-Time Guarantees for Constrained Reinforcement Learning under Partial Observability
We study reinforcement learning in constrained partially observable Markov decision processes (CPOMDPs), where an agent maximizes reward subject to a cost constraint using only incomplete information about the underlying state. The lack of state observability may require optimal policies to depend on arbitrarily long observation-action histories, making planning and learning challenging. We leverage recent advances in finite-memory approximations of POMDPs and represent finite history windows as states of a constrained MDP. We then develop a primal--dual algorithm that provides the first non-asymptotic guarantees that jointly bound average reward suboptimality and constraint violation in the original CPOMDP. A key challenge is that this finite-window approximation affects both reward values and the set of feasible policies. Under a filter-stability assumption, we derive bounds relating optimality and feasibility in the finite-state model to those in the original CPOMDP, with approximation errors that decay geometrically with the window length. Experiments in partially observable gridworlds demonstrate the effectiveness of our approach.
