用 Devin 从生产流量自动生成评测集，Rox 评测覆盖率从 43-57% 升至 72%
Rox 团队让 Devin 每周读真实对话、自动写评测、提 PR，覆盖率从手写的 43-57% 涨到 72%，做法可以直接抄。
AI 公司 Rox 不再手写评测：每周由 Cognition 的智能体 Devin 检查其产品智能体的真实对话流量，把每次失败在 Rox 完整副本中复现，并转化为评测用例，团队只需审查 Devin 提交的 PR。这套自动生成的评测集已覆盖产品中 72% 的真实使用场景，高于此前手写评测的 43-57%，且随 Devin 持续运行而每周增长。该流程是与 Cognition 和 Braintrust 合作搭建的。
How to use Devin to automate evals from production traffic. Rox @rox_ai Nobody at Rox writes evals anymore. Devin now QAs our agent every week, on real traffic. It reads real conversations with our agent, rebuilds each failure in a full copy of Rox, and turns it into an eval. We just review the PR. The eval suite already covers 72% of real usage happening in the product, vs. 43–57% for the evals we used to write by hand. And it’s growing week over week with Devin. Built with @cognition and @braintrust : 🔗 View Quoted Tweet 💬 0 🔄 7 ❤️ 16 👀 1238 📊 4 ⚡
