SkillBuffBench:用统一基准衡量 Agent Skills 在不同框架与模型下的真实增益
SkillBuffBench: Measuring What Agent Skills Contribute across Harnesses, Models, Skill Access Modes and Memory
现在每个 agent harness 都在推 Skills,但增益数据大多来自“人工挑好 Skill”的理想场景。这个基准把 Skill 访问、记忆、harness、模型四个变量拆开交叉测,还把成本和时间单独报出来。做 agent 开发、想判断要不要给自家工作流上 Skills 的人值得跑一遍看看协议设计。
SkillBuffBench 是一个新基准与配套评估协议,专门衡量 Agent Skills(可按需加载的过程性知识文件包)对编程智能体的实际贡献。它把 Skill 访问设为三级因子(无 Skill、精选 Skill、从已安装库自动调用),再与记忆开关、4 个 agent harness、多个模型交叉组合,覆盖 Anthropic、Google、OpenAI 的旗舰闭源模型和中文厂商的开源权重模型,每个模型在每款 harness 下都运行一遍。
评测任务由材料科学与生物学领域的专家贡献,连同他们实际使用的 Skills 一起纳入,任务套件会滚动扩充。每次运行由确定性验证器打分,不确定性用配对的任务级 bootstrap 报告,成本、agent 轮数和挂钟时间也单独列出而不折算进分数。
设计动机来自现实与既有评测的落差:过去证据几乎都来自“为任务手工挑选正确 Skill、智能体从零开始”的场景,而部署中智能体要从已安装库里自行选择,还常带着早期工作的记忆。本版本论文描述基准设计、评估协议与现有任务和 Skills,评估正在进行中,结果完成后将陆续补充。
SkillBuffBench: Measuring What Agent Skills Contribute across Harnesses, Models, Skill Access Modes and Memory
Agent Skills, file-based packages of procedural knowledge that a coding agent can discover and load on demand, are now shipped by every major agent harness, yet the evidence for how much they help comes almost entirely from settings in which the right Skill is hand-picked for the task and the agent starts from a blank slate. Neither holds in deployment, where an agent chooses from whatever library happens to be installed and frequently carries memory from earlier work. We present SkillBuffBench, a benchmark and paired evaluation protocol that treats Skill access as a three-level factor (no Skill, a curated Skill, and automatic invocation from an installed library), crosses it with memory (off and on), with four agent harnesses and with multiple models (the state-of-the-art proprietary models from Anthropic, Google and OpenAI, and the state-of-the-art open-weight models from Chinese providers), every model being run under every harness, and evaluates the result on research tasks contributed by domain experts together with the Skills they use, currently in materials science and biology, with the suite extended on a rolling basis. Every run is marked by a deterministic verifier; uncertainty is reported with a paired, task-level bootstrap; and cost, agent rounds and wall-clock time are reported alongside the score rather than folded into it. This version of the paper describes the design of the benchmark, its evaluation protocol and its current tasks and Skills. The evaluation is in progress: results will be added once the runs are complete, and the task suite, results and analysis will continue to be updated on a rolling basis.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Agent Skills、SkillBuffBench、智能体、基准评测,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始信息发布自一手权威信源「arXiv: Anthropic」(Tier-1 一手发布),具备高确定性参考价值。