SWE-Flux：仓库级代码执行推理基准，最佳模型仅得 37%
Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
一个新的代码执行推理基准，480 道题来自真实仓库的测试执行，最强模型才 37 分，看看你常用的模型到底会不会“跑”代码。
SWE-Flux 是一个仓库级动态执行推理基准，包含 480 个基于真实执行结果的实例，覆盖 12 个真实 Python 仓库。答案通过插桩测试执行自动采集，不依赖人工标注或 LLM 评判。对五个 LLM 的评测显示，最佳模型准确率只有 37%，模型在不变量、过程内控制流等局部行为上表现较好，但在跨过程执行和数据流推理上明显吃力。论文还用输入扰动自动生成新变体，成功率接近 90%，变体对模型来说更难。
Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
Large language models (LLMs) are increasingly used in coding tasks, but their ability to reason about code execution remains unclear. Existing repository-level QA benchmarks mainly evaluate static code understanding and often rely on LLM-based evaluation, while execution-reasoning benchmarks are mostly limited to snippets or functions. We introduce SWE-Flux, a repository-level benchmark for dynamic execution reasoning containing 480 execution-grounded instances across 12 real Python repositories, with gold answers automatically harvested from instrumented test executions rather than written manually or judged by LLMs. The benchmark covers singletest and multi-test questions over control flow, loops, program state, dataflow, exceptions, and program invariants. Evaluating five LLMs shows that this task remains challenging. The best model achieves only 37% accuracy. Models perform better on localized behavior such as invariants, intra-procedural control flow, exceptions, and simple loops, but struggle with dataflow, inter-procedural execution, precise state reasoning, and suite-level aggregation. Finally, we show that the oracle-harvesting pipeline can generate fresh benchmark variants using input perturbation. It successfully harvests valid variants for almost 90% of the selected instances, and the resulting variants are substantially more challenging for the evaluated models.
