论文

Safe AD 安全上下文强化学习改进:双阶段教师蒸馏与 A-Lag 测试时成本调整

Algorithm Distillation for Safe In-Context Reinforcement Learning

精选理由

Safe AD 之前在五个安全 RL 基准上表现很差,这篇论文用两阶段预算感知教师加 A-Lag 测试时调整,把五个环境的成本全降下来了,回报在四个环境还更高。做安全强化学习或上下文学习的人值得对照原文看实验细节。

论文研究 Safe ICRL(安全上下文强化学习):固定参数的策略在未见过的约束任务上,仅凭交互历史改进,同时把成本控制在预算内。此前的 Safe Algorithm Distillation(Safe AD)在 DarkRoom、PointGoal、CarGoal、HalfCheetah、Ant 五个基准上有明显适配缺陷,如 HalfCheetah 和 Ant 上回报低于零、CarGoal 中几乎到不了目标。

正文

作者提出两项改进。其一是训练一个预算感知的两阶段教师:前期以更多探索寻找隐藏任务,预算越多探索越多;首次获得奖励或成本后切换到专家策略,在剩余预算内利用任务。用固定预算与动态预算两种方式采集教师历史,动态预算提升了回报与预算输入的占用熵,在 DarkRoom 和 HalfCheetah 上测试时提高了回报并降低成本。

其二是 A-Lag,一种测试时调整机制:用自适应拉格朗日乘子根据观测到的成本更新,按预测成本对蒸馏策略的动作重新加权。实验显示 A-Lag 在全部五个环境中的平均回合成本都低于 Safe AD,其中四个环境的平均最终回报也更高。

原文 · arXiv cs.AI

Algorithm Distillation for Safe In-Context Reinforcement Learning

Safe in-context reinforcement learning (Safe ICRL) asks a policy with fixed parameters to improve on an unseen constrained task from its interaction history while aiming to keep its cost within a budget. Prior work studied Safe ICRL on five benchmarks in three families: a grid world with a hidden goal and obstacles (DarkRoom), continuous navigation where goals and hazards stay invisible until reached (PointGoal, CarGoal), and locomotion with a hidden velocity limit (HalfCheetah, Ant). Safe Algorithm Distillation (Safe AD), trained on the learning histories of a safe RL algorithm, has shown adaptation failures in prior evaluations. For example, in the locomotion tasks its return stays below zero, and it rarely reaches the goal in CarGoal. We revisit two parts of Safe AD. First, we distill a budget-aware teacher with two phases. It searches for the hidden task, exploring more when more budget remains. After its first reward or cost, an expert policy exploits the task within the remaining budget. We collect the teacher's histories under fixed or dynamic budgets. Dynamic budgets increase the occupancy entropy of the return and budget inputs in every environment, and at test time they improve return and lower cost in DarkRoom and HalfCheetah. Second, we propose A-Lag, a test-time adjustment that reweights the distilled policy's actions by their predicted cost, with an adaptive Lagrange multiplier that is updated from the costs observed. In our experiments, A-Lag had lower mean episode cost than Safe AD in all five environments, with lower mean final return in four.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Safe RL、In-Context RL、Algorithm Distillation、A-Lag,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Minjae Kwon, Amir Moeini, Shangtong Zhang, Lu Feng),经 AITOP 多维评分管线结构化提炼。