研究团队提出OptiFlow框架，通过最优传输学习离线强化学习中的多模态一步流策略
Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport
研究团队提出了OptiFlow框架，通过最优传输学习离线强化学习中的多模态一步流策略，解决了标准方法导致模式坍塌或过估计偏差的问题，实验效果不错。
这篇论文提出OptiFlow框架，用于解决离线强化学习中学习高效一步流策略的挑战。标准价值引导方法可能导致模式坍塌或过估计偏差。OptiFlow通过状态感知的最优传输，将价值感知的参考流策略和高效的一步策略耦合，利用价值定义蒸馏目标的优先级，同时确保动作距离成本保证几何兼容性，从而避免直接最大化批评器，实现高价值、数据支持的模式的内分布利用，减少越界发散风险。实验表明，OptiFlow能有效捕捉最优多模态行为，在多个离线RL基准上取得强性能。
Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport
Offline reinforcement learning aims to learn a policy solely from fixed datasets, which often contain multimodal action distributions. Flow policies can naturally represent such multimodal behaviors, but learning an efficient one-step flow policy remains challenging: standard value guidance often leads to mode collapse or exploits overestimation bias in out-of-distribution regions. To address this, we introduce One-step Flow policy via Optimal Transport (OptiFlow), a framework for one-step flow policy learning as a structured sample-allocation problem. OptiFlow jointly trains a value-aware reference flow policy and an efficient one-step policy, coupling their action samples through state-wise entropic optimal transport. For each state, critic-estimated values define the priority of distillation target actions, while the action-distance cost ensures geometrically compatible pairings. By avoiding direct critic maximization, our transport-guided approach enables in-distribution exploitation by anchoring the one-step policy to high-value, dataset-supported modes without the risk of out-of-distribution divergence. Experimental results demonstrate that OptiFlow effectively captures optimal multimodal behaviors and achieves strong performance across diverse offline RL benchmarks. Our code is available at https://github.com/Yonsei-DILLab/OptiFlow.
