研究：编码代理的编码器选择对任务成功率影响较小
Do models need their native harness for coding? Awesome work from our intern @melissapan on this. S...
朋友研究说，给编程助手选什么编码器其实没那么重要，成本才是关键，简单方案就行。
研究评估了7个模型在Claude Code、Codex和Pi三种编码器上的表现，发现编码器选择对任务成功率影响不大，但会显著影响成本。一个简单的编码器也能达到竞争力，原生编码器并非总是最佳选择。
Do models need their native harness for coding? Awesome work from our intern @melissapan on this. S...
Do models need their native harness for coding? Awesome work from our intern @melissapan on this. She dug into whether the harness (Claude Code vs Codex CLI vs Pi) actually moves the needle for coding agents. Turns out it matters way less than people assume. 21 model-harness pairs, real rigor. More from Arena to come. More details on the Arena blog: arena.ai/blog/coding-ag… Melissa Pan @melissapan Does your Claude model really need Claude Code…? 🤔 We evaluate 7 models on Claude Code, Codex, and Pi. Three surprising findings emerge: 1️⃣Harness choice has little effect on task success rate, but can significantly affect the cost 2️⃣A simple harness can be competitive 3️⃣The native harness isn’t always the best. Millions of people are using coding agents, but the impact of harness choice remains unclear. (1/n) More details in the thread. 🧵 🔗 View Quoted Tweet 💬 6 🔄 4 ❤️ 36 👀 3770 📊 9 ⚡
