论文提出 DS-QA 基准与蒙特卡洛协议评估 LLM 数据科学任务可靠性
One Run Is Not Enough: A Monte Carlo Protocol for Evaluating LLM Reliability on Data Science Tasks
这篇论文做了件常被忽略的事：同一题让模型跑 100 遍看答案稳不稳，结果数据可视化和监督学习这类任务波动最大，做评测的人该看看。
论文介绍 DS-QA 基准，包含 100 道来自英国大学课程的数据科学与统计题目，覆盖 16 个数据集和 3 个难度等级。评估框架分三层：通用响应属性（运行时间、冗长度、文本相似度）、粗粒度指标（准确率、代码可执行性、完整性）和任务特定指标。团队用蒙特卡洛协议对 OpenAI 模型重复运行 100 次，发现数据可视化与监督学习任务在中高难度下响应变异性显著。论文还测得词冗长度比每增加 0.1，token 冗长度比约增加 0.013。
One Run Is Not Enough: A Monte Carlo Protocol for Evaluating LLM Reliability on Data Science Tasks
Large language models (LLMs) have shown promise for automating data science workflows. Benchmarks such as DataSciBench (2026) and StatQA (2024) have advanced the evaluation of their capabilities across data science and statistical tasks. Assessing their reliability requires examining both the quality of their responses and their consistency across repeated runs, task types, and difficulty levels. With this aim, we introduce DS-QA, a benchmark comprising 100 data science and statistics questions drawn from UK university courses, spanning 16 datasets and three difficulty levels. We develop a three-layer evaluation framework that combines general response properties (runtime, verbosity and text similarity), coarse-grained metrics (accuracy, code executability and completeness), and task-specific metrics. We evaluate OpenAI models using a Monte Carlo protocol with 100 repeated runs. The results reveal substantial variability in responses to data visualisation and supervised learning tasks, particularly at medium and hard difficulty levels, highlighting the importance of evaluating consistency alongside response quality. We also find that an increase of 0.1 in the word verbosity ratio corresponds to an increase of about 0.013 in the token verbosity ratio.