200道题即可复现智能体评测满分差1.03分的方法
测一个线上Agent要跑几千道题太贵？这篇论文用200道题就能把分数误差压到1分左右，实测还迁移到了5个不同Agent上。
一项研究提出用200道题（占完整基准的38.5%）重新评估生产环境智能体，误差控制在1.03分以内。作者分析了一个月活数万用户的分析智能体，基于574次历史评测运行，按日期切分校准期与留出期。对比随机抽样、缓存结果、固定子集与基于项目反应理论的自适应测试后，多维2PL自适应测试保真度最高。团队最终部署了按难度分层的固定子集，因其更易运行，且该子集无需重新校准即可迁移到另外5个智能体家族，校准窗口短至1天仍保持稳定。
Nice paper showing how to re-evaluate a production agent at a fraction of the cost. 200 questions, 38.5% of the full benchmark, reproduce the full score to within 1.03 points. The authors studied an analytics agent that serves tens of thousands of monthly users, using 574 historical benchmark runs split by date into calibration and held-out periods. They compared random sampling, cached results, fixed representative subsets and adaptive testing based on item response theory. Multidimensional 2PL adaptive testing gave the best fidelity. The team deployed difficulty-stratified fixed subsets instead because they are simpler to run. Those subsets transferred to five other agent families without recalibration and stayed stable with calibration windows as short as one day. Paper: arxiv.org/abs/2609.21267 Chat with Paper: academy.dair.ai/papers/efficie… 💬 2 🔄 2 ❤️ 4 👀 1605 📊 4 ⚡
