论文用六维文化回应标准审计 SimpleQA 与 Chatbot Arena
Whose Facts Count? A Culturally Responsive Audit of LLM Evaluation Benchmarks
SimpleQA 全部题目只认英文档案，Arena 七成多提问是英文，这篇论文把评测偏差算成了具体数字。
一篇 arXiv 论文借用文化回应性评估（CRE）框架，用六个维度审计了 OpenAI 的 SimpleQA（4,326 题）和 LMSYS Chatbot Arena（600 段对话）。
SimpleQA 的全部 4,326 道题都以英文档案文献作为验证依据，且仅一位标注者对哥伦比亚建国年份的偏好就占全部题目的 2.70%，虚增了全球南方覆盖的表象。
Chatbot Arena 中英文提示词占 76.3%，而国际电信联盟（ITU）估计英文用户仅占全球网民的 25.
9%。论文构建的 50 题反向基准测得 SimpleQA 的文化回应性缺陷均值高出近三倍（Cohen's d = 1.
01），并据此提出一套可操作的 CRE 评估框架。
Whose Facts Count? A Culturally Responsive Audit of LLM Evaluation Benchmarks
LLM benchmarks function as evaluation instruments, informing decisions that affect education, labor, and public services worldwide. Drawing on Hood, Kirkhart, and Hopson's culturally responsive evaluation (CRE) frameworks, this paper applies a six-dimension CR rubric to audit OpenAI's SimpleQA (N = 4,326 items) and the LMSYS Chatbot Arena (N = 600 conversations). Every SimpleQA question requires English-language archival verification as its evidentiary basis. A single rater's preoccupation with Colombian founding dates accounts for 2.70% of items, inflating the appearance of Global South coverage. English-language prompts constitute 76.3% of Arena conversations, against an International Telecommunication Union (ITU)-estimated 25.9% share of global internet users. A 50-item counter-benchmark scored a mean CR deficit nearly three times lower than SimpleQA (Cohen's d = 1.01). The paper proposes a practical CR evaluation framework. These are structural validity failures, not incidental measurement problems, with direct consequences for communities whose knowledge traditions these instruments were not built to see.
本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「中美大模型技术代差与商业化对照」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：OpenAI、SimpleQA、Chatbot Arena、基准评测，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始信息发布自一手权威信源「arXiv: OpenAI」（Tier-1 一手发布），具备高确定性参考价值。截至目前，全网已有 8 家独立信源对该事件进行交叉印证与跟进报道（含 AI Will、掘金本周最热、Decoder 等），属于近期高关注度行业信号。
