Kaggle 举办 Bench Summit，探讨基准测试失效的四大原因
Kaggle 开了个峰会专门聊 benchmark 为什么不可信，列了 4 个坑和 4 个修法，做模型评测的值得看看他们怎么想。
Kaggle 主办的 Bench Summit 峰会开幕，Janie Gu 分享了基准测试现状的四个问题：与生产环境不相关的 relevance、一个发布周期内就饱和的 eval、各家实现不同导致分数波动大的 verification，以及难搭建的 eval 基础设施。她提出四个修复要点：以真实工作流为基础、用众包任务和 arena 构建持续评测、由社区专家审计的透明方法论、开箱即用的 eval 基础设施。
Bench Summit by Kaggle has officially kicked-off!
Trust in benchmarks is eroding. The path forward starts here.
We started strong with @janiejgu sharing key insights on the problems with benchmarks today and Kaggle’s mission to build trust in evals and help the world discover what works and doesn’t work in AI.
During the session, she shared four ways benchmarks fail today: 1. Relevance: Many public benchmarks don’t resemble what your agents do in production. 2. Saturation: Evals go stale within a release cycle. 3. Verification: Everyone has their own implementation of the same benchmarks, meaning scores vary wildly due to infrastructure, not model quality. 4. Infrastructure: Evals infra is hard to build correctly, and everyone rolls their own.
To get numbers you can actually trust, Janie highlighted four essentials: 1. Real workflows as the foundation for benchmarks 2. Continuous benchmarks from crowdsourced tasks, arenas, and production data 3. Rigorous and transparent methodology audited by the community experts 4. Eval infrastructure that everyone can use out of the box
Here’s a sneak peak of the early moments:
