论文

论文提出测量语言模型时间数据污染的新方法

Reading or Remembering? Measuring Temporal Contamination in Language Models

精选理由

做经济预测或时间敏感评估的人都该看看这篇。它发现只给日期模型就能拿到 0.64 AUC,随机基线会高估模型能力。文中给出的无记忆模型对比方法可以直接搬进自己的评测流程,尤其是金融和宏观研究方向。

一项研究指出,用语言模型解读美联储 FOMC 声明做经济预测时存在时间维度的测试集污染:模型训练语料晚于所读文档,可能只是记住了 2022 年通胀上升等结果。

正文

研究覆盖 2000 至 2026 年的 FOMC 声明与会议纪要,将现代开源模型与仅在固定日期之前文本训练的时间一致模型对比。结果显示,仅凭日期和时代先验,模型就能在无法知晓答案的 GDP 问题上达到 0.

64 的 AUC,说明随机基线并不适用;而与同题目的无记忆模型对比时,多数现代模型能回忆出哪些年份增长强劲。

成员推断得分按时代和文档类型排序,而非按模型是否见过该文档分类,研究用开放预训练语料中的精确计数验证了这一点。在美国 GDP 的实时即时预测中,所有被估计的前瞻溢价都低于宏观样本量可检测的水平,GDP 与通胀上的结果经多重检验校正后无一存活。

研究结论是:前瞻性审计应在相同题目上用无记忆模型直接测量记忆。

原文 · arXiv cs.LG

Reading or Remembering? Measuring Temporal Contamination in Language Models

Language models are now used to read central-bank text for economic forecasting, but most were trained on text written after the documents they read. A model reading a 2021 FOMC statement may already know that inflation rose in 2022, so part of its apparent skill can be memory of the outcome, a temporal form of test-set contamination. We measure this memory on FOMC statements and minutes from 2000 to 2026 by comparing modern open models with chronologically consistent models, which are trained only on text from before a fixed date. Asked about past GDP outcomes given only a date, models that cannot know the answer still reach an AUC of up to 0.64 from priors on dates and eras, so chance is the wrong baseline. Against these memory-free models on the same questions, most modern models recall which years had strong growth. Membership-inference scores sort documents by era and document type rather than by whether a model saw them, as exact counts in an open pretraining corpus show. In real-time nowcasts, every lookahead premium we estimate on US GDP is smaller than the forecast comparison can detect at macroeconomic sample sizes, and none on GDP or inflation survives correction for multiple testing. Lookahead audits should therefore measure memory directly against memory-free models on the same items.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:数据污染、评估基准、FOMC、GDP预测,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Arunkumar V, Gangadharan G. R., S. Senthilkumar),经 AITOP 多维评分管线结构化提炼。