Jerry Liu谈基准测试分类：前沿智能与专用智能各有侧重

There are two main types of benchmarks: 1. those that measure frontier intelligence (terminalbench, gdpval, etc.). accuracy matters more than cost. 2. those that measure specialized intelligence. the pareto curve of cost/accuracy more than accuracy alone. both are important, but they are distinct benchmark categories. * as (1) saturates, they become (2). * the largest frontier models at xhigh likely already saturate (2), but they do so at a cost/latency point that's completely price-prohibitive. * every user should define (2) for the tasks they care about the most. we're big fans of both. we have so far dedicated outsized attention to (2) especially related to document tasks. we fundamentally think users should have the most price-efficient tools to understand, extract, and read over their documents. we've built parsebench, extractbench for this purpose. we are starting to invest in (1) as well, because certain document tasks are so challenging that even frontier models struggle (e.g. see extractbench pro). we are continuing our investment in long-horizon workflow research at @llama_index 💬 5 🔄 0 ❤️ 8 👀 1010 📊 7 ⚡