System1-Arena 基准：哪些决策可以交给 System 1 模型
System1-Arena: Which Decisions Can Be Left to System 1?
这篇论文测了 30 个模型在 92k 道决策题上的表现，告诉你什么时候该用一步出概率的 System 1 模型，什么时候必须上推理模型，延迟和成本差得很实在。
arXiv 论文提出 System1-Arena，一个评估 System 1 类模型（如 Jev 和 OpenAI Decisions API）直接输出决策概率能力的基准。该基准包含 92k 道题、105 个任务，数据来自 268 个公开来源，另含 138 个闭环环境，覆盖网页购物、SWE-bench 编程控制、机器人、游戏和驾驶。评测了 30 个模型，其中 7 个闭源 LLM、13 个开源 LLM、10 个 System 1 模型。结果显示最大的 System 1 模型在非智能体任务上的准确率匹敌或超过快速闭源 LLM，且概率校准更好，延迟只有四分之一；推理型 LLM 准确率最高，但成本约为快速 LLM 的 70 倍。闭环环境中大型开源 LLM 成功率最高达 32.9%，小型 System 1 模型很少成功。
System1-Arena: Which Decisions Can Be Left to System 1?
Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.
