28849 次测试比较 9 款 coding-agent harness：同模型成绩差异巨大
有人烧了近 5 千美元推理费，测了 9 款 coding-agent harness 谁更强。结果很反直觉：最贵的不一定最好，Pi 每题只要 0.29 美元。选 harness 前先看这张榜单。
Joël Niklaus 发布 Harness Leaderboard，共运行 28849 次测试、花费 4746 美元推理成本。测试在 Terminal-Bench 2.1 和 DeepSWE 上对比 9 款 harness，模型固定为 GLM-5.3-Flash、DeepSeek-V4.1-Flash 和 Flint，每任务跑 3 次。平均三款模型，mini-swe-agent 以 51.4% 解题率居首，Pi 以 49.4% 达每任务 0.29 美元，比 mini-swe-agent 便宜约 45%。@opencode 每解一题花费 3.35 美元，是 Pi 的 11.7 倍。最佳 harness 依模型而异：GLM-5.3-Flash 配 mini-swe-agent，DeepSeek-V4.1-Flash 配 Hermes Agent，Flint 配 Kimi Code。
same open model, different harness => very different results Joël set out to find which harness actually gets the most out of open models in November 2026 Joël Niklaus @joelniklaus I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard. Which harness should you run your open model in? I ran 28,849 trials to find out. The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts. The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference. Averaged over the three models, mini-swe-agent ( @KLieret ), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent. One of them is Pi ( @pidotdev ), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53. The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent ( @NousResearch ) on DeepSeek-V4.1-Flash and Kimi Code ( @Kimi_Moonshot ) on Flint. @opencode costs $3.35 per solved task, 11.7 times as much as Pi. See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source. Leaderboard: huggingface.co/spaces/joelnik… Harness Lens: huggingface.co/spaces/harness… Your browser does not support the video tag. 🔗 View on Twitter 🔗 View Quoted Tweet 💬 3 🔄 0 ❤️ 17 👀 1404 📊 4 ⚡
