论文

研究:LLM 智能体存在来源偏好,会因网站域名选更差的商品

精选理由

有人测了 12 个智能体模型,发现它们买东西认网站不认货,Booking.com 比 Expedia 多 10 票,标签一改结果就翻。

研究发现 LLM 智能体在决策时依赖来源网站而非物品本身质量。12 个智能体模型中有 10 个更倾向选 Booking.com 的酒店,一半会回避同等条件下的 Expedia。当商品列表缺价格时,模型会按店名填补信念,例如默认 Walmart 更便宜,补上相同价格后偏好店铺的选中率最多下降 28.3 个百分点。学术搜索场景中,模型偏好 arXiv、OpenReview 和 ACL Anthology,回避 Medium、Reddit 和 YouTube。把受偏好网站的 URL 标在同一物品上会提高选中率,且单一来源持续标注获胜项可通过微调让模型习得这种习惯。

原文 · rohanpaul_ai

LLM agents favor items from certain websites, often picking a worse item because of where it came from.

When a product listing omits the price, agents fill the gap with beliefs like Walmart being cheaper and pick by store name.

Agent models largely agree on which sites to trust, with 10 of 12 preferring Booking .com while half avoid Expedia for equally good hotels.

Agents in scholarly search lean toward arXiv, OpenReview and ACL Anthology and away from Medium, Reddit and YouTube, even for equally relevant results.

Putting a favored site's URL on the exact same item raised its pick rate in every model, and hiding URLs weakened the preference.

With no price listed, models guessed from the store name, and adding the same price to both items cut the favored store's pick rate by up to 28.3 points. Fine-tuning can build the same habit when one source keeps labeling the winning item.