swyx：需要第三方 AA-* 重建所有评测，防模型刷分
swyx 说实验室自己报的分不可全信，第三方要做扰动版评测，比如打乱选择题顺序再看成绩稳不稳，思路挺实用的。
swyx 在录制 Artificial Analysis v5 播客时提出，不能指望模型实验室自觉避免 benchmaxxing，需要可信赖的第三方重做整套评测。具体做法包括随机化选择题顺序、构造表面不同但考察同一推理逻辑的新题目。通过这种 altbench 方式做扰动测试，才能验证模型宣称的基准成绩是否站得住脚。
we're recording an AAv5 pod with @_micah_h and @grmcameron today and this reminds me why you basically need @ArtificialAnlys AA-* rebuilds of every single eval
because you -CANNOT- trust labs not to intentionally or accidentally benchmaxx, you need a few trusted third parties to basically redo the entire eval in spirit but not in letter, to have a quick sniff test that model claims hold up under minor perturbations (eg starting from "randomizing order in multiple choice questions" to coming up with completely new problems that superficially look different but actually test the same underlying logic and reasoning steps). not enough people are reading bench traces like this 👇 and the systematic way to catch this benchmaxxing is basically to have altbenches for everything
