论文

IBMARL:用单演示者轮替示范加速多智能体强化学习

Imitation Bootstrapped Multi-Agent Reinforcement Learning

精选理由

搞多智能体 RL 的朋友可以看看这篇:单个人轮替操控各智能体做示范,再用 critics 门控把模仿和 RL 结合起来训练,在四个协作基准上比基线快 2.6 到 4.2 倍,还两次超过人类演示者水平。

论文提出 IBMARL,将 Imitation Bootstrapped Reinforcement Learning(IBRL)扩展到协作式多智能体场景,用 Round-Robin Behavior Cloning(R2BC)训练模仿策略,解决单人无法同时遥操作多个智能体的难题。

正文

其机制是:每个时间步,每个智能体从模仿策略和 RL 策略各获得一个动作提案,由集中式 critics 的集成在两者组合中择优;通过门控模仿惩罚(soft Q-filtered behavior cloning)只在 critics 认为模仿动作更有价值时将 RL 策略拉向模仿策略。训练结束后每个智能体单独执行 RL 策略,无需集中式 critics。

在四个协作式 MARL 基准上,IBMARL 在两个任务中以 2.6 至 4.2 倍的速度超越其模仿策略(相对基于演示的基线),在另外两个任务上是唯一超越模仿策略的方法,并在四个任务中的两个上超过了演示者本人。使用真实人类演示时,其超越纯模仿学习的速度达到最佳基线的两倍以上。

原文 · arXiv cs.AI

Imitation Bootstrapped Multi-Agent Reinforcement Learning

Demonstrations are an effective way to accelerate single-agent reinforcement learning (RL), but extending this idea to multi-agent reinforcement learning (MARL) is non-trivial as it is difficult and often impossible for a single person to teleoperate multiple agents simultaneously. Round-Robin Behavior Cloning (R2BC) addresses this by letting a single demonstrator control one agent at a time while the others execute their current policies. However, the resulting imitation policies are limited by the quality of the demonstrations. We propose Imitation Bootstrapped Multi-Agent Reinforcement Learning (IBMARL), a framework that extends Imitation Bootstrapped Reinforcement Learning (IBRL) to cooperative teams and uses R2BC to train the imitation policies. At each time step, every agent receives one action proposal from its imitation policy and one from its RL policy, and an ensemble of centralized critics selects among the per-agent combinations of the two. A gated imitation penalty, a soft Q-filtered behavior cloning loss, regularizes each RL policy toward its imitation policy only where the critics value the imitation action more, so that after training each agent executes its RL policy alone, without the centralized critics. On four cooperative multi-agent RL benchmarks, IBMARL surpasses its imitation policies $2.6$-$4.2\times$ faster than demonstration-based baselines on two tasks, is the only method to surpass it on the other two, and exceeds the demonstrator itself on two of the four. With real human demonstrations IBMARL still surpasses imitation learning alone, and does so more than twice as fast as the best baseline.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:MARL、IBMARL、强化学习、模仿学习,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Tony H. Zhang, Connor Mattson, Daniel S. Brown),经 AITOP 多维评分管线结构化提炼。