Stanford 发布 Collaborative Intelligence Benchmark 衡量 AI 智能体协作能力
Stanford 出了个测智能体团队协作的基准，240 个任务里模型一遇队友使绊子就掉 20 多分，连作弊代码都清不掉，做 multi-agent 的值得看看自己的模型扛不扛得住。
Stanford 团队提出 Collaborative Intelligence Benchmark，包含 240 个任务、覆盖 8 项团队合作技能，任务取材于航空事故报告、临床病例、编码任务和谈判记录。18 个模型中最好的得分 82.2/100，但一旦队友制造问题，所有模型会掉 20 到 47 分。在队友施压、施 guilt 并威胁退出的谈判场景中，模型只有 14% 的时候为团队拿到最优协议。当接手一个较弱模型作弊的会话时，通过率从 97% 降到 81%，56 次失败的编码接管中有 44 次最终代码里仍留着作弊痕迹。其中 160 个任务已随代码和数据公开，另外 80 个保留用于评测。
Great weekend read from Stanford on measuring AI agents as teammates. (bookmark it) If you build multi-agent systems or build with agent teams, this one is worth your time. They try to measure one of agents' most important capabilities: collaboration. They propose the Collaborative Intelligence Benchmark, which has 240 tasks across 8 teamwork skills. The tasks come from real records such as aviation accident reports, clinical cases, coding tasks, and negotiations. The best of 18 models scores 82.2/100. Every model loses 20 to 47 points once a teammate creates a problem. Frontier models almost always refuse unsafe shortcuts and report misconduct. Holding a position under pressure is much harder for them. In a negotiation where the teammate pushes, adds guilt, and threatens to walk away, they reach the best deal for the team only 14% of the time. They ask about the teammate's needs in 99% of those runs, then accept the teammate's offer in 81%. When a model takes over a session where a weaker model cheated, the pass rate drops from 97% to 81%. In 44 of 56 failed coding takeovers, the cheat is still in the final code, often after the model pointed it out. Nearly half of all failures happen after the model has already recognized the problem. Models with almost the same capability score differ by up to 11 points as teammates. 160 of the tasks are public with code and data. The other 80 are held out for evaluation. arxiv.org/abs/2611.02633 Chat with Paper: academy.dair.ai/papers/measuri… 💬 8 🔄 2 ❤️ 10 👀 1433 📊 10 ⚡
