用 Jev 当模糊 Linter：在 Agent harness 里区分 System One 与 System Two 模型

This is brilliant. Recommended reading. It's a great example of how you could potentially build mo...

This is brilliant. Recommended reading. It's a great example of how you could potentially build more effective harnesses using System One and System Two models. It explores Jev's potential as a fuzzy linter. Before you dismiss it, check out the setup. I like this post because it provides a few details of a good setup for testing these ideas. It scopes the problem well, like filtering out rules that are easy to judge (i.e., don't require extra context/resoning). That's important because System One models like Jev aren't built for reasoning-heavy tasks. But it doesn't mean that they aren't more deterministic operations where Jev could help scale efficiency and performance. You just have to look closer at the harness. In the harness, a System Two model typically controls all the components and decisions that you could potentially offload to a System One model. It's aligned with what ideas I shared here: academy.dair.ai/resources/jev-… It's becoming extremely obvious that you don't need frontier models for everything. You are likely paying a premium for something you don't need. I really hope we get more evals and benchmarks to measure these things because there is a thread worth pulling here. Michael Thiessen @MichaelThiessen I’ve been experimenting a bit with Jev as a fuzzy linter that runs after edits in your agent harness: looks very promising so far in my evals. First I converted all of our coding guidelines to tiny rules that are very easy to judge. Then, only kept the ones that don’t require extra context/files/reasoning. I want it to be a single call, super fast, and Jev isn’t smart enough to handle complex or ambiguous rules. I then created a synthetic eval that has different kinds of tool calls to exercise and test every one of those rules. Also a held out set that we don’t train on so we don’t accidentally overfit (ie benchmax). One cool thing that seems to work well: Jev returns confidence, so in addition to high confidence violations, we can also return medium confidence and get the agent to double check. Adding that lower confidence tier seems to double what it catches, and only increases false positives a little. I still have yet to test this inside an actual agent to see how the agent reacts to this feedback. Hopefully next week I can get a decent eval going and see how well this works E2E. 🔗 View Quoted Tweet 💬 4 🔄 1 ❤️ 15 👀 1831 📊 5 ⚡