研究测试 75 道“防 AI”大学考题,最强 LLM 仍答对约 83%
Generative AI and the Crisis of Higher Education Assessments: Examining 'AI-Proofing' Strategies for College-Level Assessments
37 位大学老师拿出了他们自信防得住 AI 的考题,结果最好的模型还是答对 83%。文章把老师们的出题套路总结成五种策略并逐一验证效果,想改考试设计的大学老师和学生都该看看,比一般空谈学术诚信的论文实在得多。
arXiv 上的一项研究收集了 75 道真实大学课程考题,覆盖 17 个学科,由来自十所美国大学和一所英国大学的 37 名教师提供,其中 61 道被教师刻意设计为“LLM-proof”,14 道来自在线课程。
研究者用三个最新 LLM 在有无课程补充材料两种条件下作答,表现最好的模型无材料得分约 83%,有材料得分约 85%。通过主题分析,研究归纳出教师用来增加难度的五种策略,并检验这些策略是否真正影响模型准确率。
结果显示策略之间存在明显效果差异,且不同模型、不同答案格式的表现差别可观。研究指出,多数所谓防 AI 的题目设计并未真正挡住 LLM,这对高校学术诚信政策与考核设计有直接参考价值。
Generative AI and the Crisis of Higher Education Assessments: Examining 'AI-Proofing' Strategies for College-Level Assessments
Higher education is confronting an academic-integrity challenge unlike any in its history. Students can now outsource homework, take-home assignments, and online examinations to large language models (LLMs) that complete them in seconds. Existing research on evaluating LLM performance on academic assessments has focused largely on standardized examinations, or on questions from college-level courses drawn from a single discipline. Far less is known about how LLMs perform on college-level assessment questions spanning multiple disciplines, that instructors have purposely designed to be AI-resistant, and about which features of such questions, if any, meaningfully constrain LLM performance. We address this gap by collecting 75 questions from authentic college coursework assessments, spanning 17 disciplines, which were contributed by 37 faculty members from ten U.S. universities and one U.K. university. Of these, 61 questions were intentionally designed by faculty members to be 'LLM-proof', while the remaining 14 questions were sourced from online courses. We then (i) evaluate three state-of-the-art LLMs' ability to correctly answer these questions with and without supplementary course materials; (ii) identify five strategies, through thematic analysis, that faculty used to make their questions harder for LLMs to solve; and (iii) test whether these strategies impact LLMs' ability to answer the questions accurately. Our experimental results show that the best-performing model scored approximately 83% without supplementary course materials and 85% with them, and we observed noteworthy differences across models, answer formats, and instructor tweaking strategies. We discuss implications for assessment design, academic-integrity policy, and the broader ethics of deploying LLMs in educational settings.
本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。涉及核心实体与技术议题包括:LLM、学术诚信、高等教育、评估基准,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Bonam Mingole, Davis Yadav, Hui Min Lee, S. Shyam Sundar, Amulya Yadav),经 AITOP 多维评分管线结构化提炼。