哈佛-MIT 论文:让 Agent 提前规划反而降低博弈表现
哈佛-MIT 用拍卖实验发现,让 Agent 先做计划这招反而拖后腿,改界面才管用,做 Agent 的人该看看
哈佛-MIT 的一项研究在拍卖与匹配市场环境中测试“提前规划”类提示词的效果,这些环境存在已知最优策略,可以逐项给决策打分。结果显示此类提示词让模型整体表现变差。真正有效的是改交互界面:升序拍卖每次只呈现一个安全选项,在四个模型家族中降低了出价错误率;直接说明收益和真实出价为何安全也有效。此外,Agent 陈述的计划与其实际选择并不对应,改善出价的干预措施并未提升计划中的推理质量。
Does telling your agent to plan ahead actually help?
It's standard practice to add a "plan ahead" or "think about the other players" instruction to an agent's prompt.
In auctions and matching markets, this Harvard-MIT paper finds that those prompts make play worse overall.
The authors use settings with known optimal strategies, so they can score every choice.
What helped was changing the interface.
An ascending auction, which presents one safe choice at a time, reduced bid errors across four model families. Plain descriptions of the payoffs and of why truthful bidding is safe also helped.
The agents' stated plans did not track their choices.
Interventions that improved bids left the measured reasoning quality in the plans unchanged, and some prompts improved the plans without improving bids.
The takeaway here is to evaluate a scaffold on the decisions it produces.
Paper: https://t.co/dILM8pgzbu
Chat with Paper: https://t.co/9NPg5JzTXS