Artificial Analysis 发布 Endpoint Accuracy Index v1.1
这个榜单专查各家推理服务商有没有偷工减料，四分之一端点准度不达标，还点名了截断和工具调用出错的 provider，选 API 前先看一眼。
Artificial Analysis 推出 Endpoint Accuracy Index v1.1，覆盖 100 多个 serverless 推理端点，约四分之一的端点得分低于官方权重的自部署基准。评分基于三个等权测试：BFCL-500（工具调用）、HLE-250（科学推理）和 AA-LCR-25（长上下文召回），其中 AA-LCR 升级到 v1.1 并将重复次数从 10 提到 25，置信区间从 ±4.7 收窄到 ±2.6。GLM-5.3 Flash 的 22 个端点中 16 个与基准持平，但有端点因 32K 输出 token 截断导致 HLE-250 得分跌至 22%-29%，而参考值为 55%。Muse Spark Open 的 5 个端点中 3 个达标，另两个在 BFCL-500 上把工具定义的文本参数错误返回为数字，得分 50%-51%。
Launching Artificial Analysis Endpoint Accuracy Index v1.1: a minor version upgrade alongside expanded coverage for GLM-5.3 Flash and Muse Spark Open. Across the 100+ serverless endpoints we benchmarked, nearly 1 in 4 score below the reference
Providers trade off accuracy to optimize for speed and cost. They quantize weights, write custom kernels, tune their inference stacks and cap output lengths, and sometimes they simply ship bugs. The Endpoint Accuracy Index measures how much of a model’s accuracy each endpoint preserves, so developers can pick providers on accuracy, not just price and speed.
We score each endpoint as a percentage of our self-hosted reference deployment of the official weights, across tool calling (BFCL-500), scientific reasoning (HLE-250) and long context recall (AA-LCR-25), equally weighted.
What’s changed:
➤ Upgraded AA-LCR-25 to AA-LCR v1.1 and increased repeats from 10 to 25: AA-LCR v1.1 corrects errors and ambiguities in answer keys and moves grading to GPT-5.6 Luna (medium). Running each question 25 times, up from 10, reduces the average 95% confidence interval from ±4.7 to ±2.6 points
➤ Expanded coverage to six open weights models: We add GLM-5.3 Flash (max) and Meta’s Muse Spark Open (xhigh), alongside DeepSeek V4.1 Flash 1017 (max), GLM-5.3 (max), Kimi K3 (max) and gpt-oss-120b (high)
Key takeaways for GLM-5.3 Flash:
➤ 16 of 22 endpoints are statistically in line with the reference, and one scores above it (104.2)
➤ Output token caps limit the model on complex workloads: Two endpoints capped at ~32K output tokens cut off about half of HLE-250 responses before an answer, scoring 22% and 29% against 55% for the reference
➤ Endpoints that reason less give lower quality answers: Two endpoints produce ~30% fewer reasoning tokens than the reference on HLE-250, with no truncation, and score 48% against 55%
Key takeaways for Muse Spark Open:
➤ Three of five endpoints match the reference: They score 99.5 to 99.9, including one serving FP4 weights
➤ The other two lose accuracy only on tool calling: Both return tool arguments that the tool defines as text, such as booking IDs, as numbers, scoring 50% and 51% on BFCL-500 against 59%. On HLE-250 and AA-LCR-25 they are within 2 points of the reference
- Anyscale11-05 17:00原文
