用多智能体强化学习在 Polymarket 订单簿上训练做市商
Cooperative Multi-Agent Reinforcement Learning for Market Making on Prediction-Market Order Books
有人在 417 个 Polymarket 真实市场上用强化学习训练做市机器人，还拆成报价和风控两个角色，方法细节和数据重建都写得很清楚。
研究以 Polymarket 预测市场为对象，从 417 个真实市场重建订单级 LOBSTER 格式数据，在 JaxMARL-HFT 环境中训练做市智能体。作者将任务拆分为报价器和风险管理器两个角色，用循环 IPPO（RNN 策略）训练，并通过奖励混合和 post-fill markout 缓解信用分配难题。重构后的单智能体做市商比重训的 JaxMARL-HFT 基线报价更频繁、持有期损失更小。纯角色分解效果不佳，配合奖励设计才取得最好结果。不过在周期性留存集评估中，没有任何配置实现正的平均组合收益。
Cooperative Multi-Agent Reinforcement Learning for Market Making on Prediction-Market Order Books
A market maker provides liquidity by quoting prices to buy and sell an asset, earning the spread while managing inventory risk. We study market makers trained with multi-agent reinforcement learning on prediction markets, specifically Polymarket, whose thin order books pose distinct challenges. Three challenges underpin this research: adverse selection may cause a policy trained on real data to stop quoting; the consequences of a quote can arrive many steps after the action; and prediction-market L2 data does not identify individual orders needed for simulation. We address these challenges by reconstructing order-level LOBSTER-format data from Polymarket and training agents in JaxMARL-HFT on 417 real markets. We enrich the baseline market maker's observation space with the book and recent flow, redesign its actions and inventory controls, and propose a cooperative multi-agent decomposition into a quoter and a risk manager trained using recurrent Independent Proximal Policy Optimisation (IPPO) with RNN policies. We investigate reward shaping, particularly reward blending and post-fill markout, to mitigate credit-assignment difficulties between the agents. Our redesigned single-agent market maker quotes more frequently and incurs smaller held-out losses than the retrained JaxMARL-HFT baseline, although the contributions of jointly changed observations, actions and risk constraints are not separately identified. Decomposition alone performs poorly; careful reward design substantially improves the desk relative to the unshaped split and yields our best result. Models are evaluated on realised end-of-episode portfolio value on held-out markets. None establishes positive average portfolio value in periodic held-out evaluation, and each configuration was trained with one seed. The environment, data and protocol lay a foundation for makers that quote many linked markets at once.
