arXiv 论文：LLM 能否识破造假的量化回测？
Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark
量化圈的朋友看过来，这论文测了 LLM 审回测靠不靠谱，DeepSeek 召回满分但一不留神就把好策略全打假，方法设计挺有意思。
arXiv 论文构建了一个含 96 组配对样本的回测审计基准，每个有缺陷的回测都配有仅改动一处方法细节的干净对照策略。在四个文本端点共 1440 次缓存审计中，DeepSeek 审计器对闭式与 clean-aware 代码缺陷的召回率达 100.0%，但开放提示词会将 93.8% 的干净代码误判为有缺陷。加入 clean-aware 警告后，DeepSeek 代码误报率从 20.8%（95% CI 11.7–34.3）降至 0.0%，召回率不变。仅报告召回率时四个模型中有三个排名相同，而引入干净对照误报率后差距拉开 79 个百分点。
Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark
Backtest auditing is a calibration problem: high flaw recall is not useful when the model falsely flags matched clean strategies. We build a 96-item paired benchmark in which every flawed backtest has a clean control that holds strategy, dates, code style, labels, and reporting scaffold fixed while changing one methodology detail. A deterministic scorer separates flaw recall, clean-control false positives, evidence localization, and fix relevance. Over 1440 cached audits from four text endpoints, the primary DeepSeek auditor reaches 100.0\% closed and clean-aware code recall, but open prompts over-flag 93.8\% of clean code controls, and clean-aware all-three specificity is 87.5\% even where recall saturates. A clean-aware warning drops DeepSeek code false positives from 20.8\% (95\% CI 11.7--34.3) to 0.0\% (0.0--7.4) at unchanged recall, while the budget anchor still flags 38/48 clean controls under the same prompt. Reporting recall alone would rank three of these four models identically; reporting the clean-control rate separates them by 79 points.
