论文

LECAT 框架揭示预训练数据如何影响 LLM 类别知识学习

LECAT: Measuring LLM Category Learning from Pretraining Data

精选理由

做预训练或数据清洗的人应该看:研究者从零训了 68M 小模型,证明语料里逐字重复的规则式句子反而让模型学不会灵活泛化,混合多种表达方式效果更好。对大模型也一样成立,能直接指导数据配比。

论文 LECAT 提出一个两部分框架,评估 LLM 学习类别知识的三种能力:记住训练域内的类别知识、泛化到未见实例、以及在合适语境中灵活运用知识。

正文

作者用 199 个精选的类别-特征对测试开源预训练模型,并分析其训练数据,发现上下文增强和类别-特征共现概率(PMI)越高,大模型学习效果越好,而逐字重复的规则式陈述会削弱灵活的类别学习。为隔离这些因素,团队从零训练了一个 68M 参数模型,精确控制其训练数据中的类别信息。

实验确认逐字重复会降低灵活学习能力,而混合多样化的知识表达形式则带来更稳健的类别学习。结论指出,语境灵活性对模型始终困难,类别知识在预训练语料中的表达方式显著影响模型获得稳健知识。

原文 · arXiv cs.AI

LECAT: Measuring LLM Category Learning from Pretraining Data

Humans know that airplanes are fast, that this extends to red airplanes, but that it does not hold for broken airplanes. How do LLMs acquire category knowledge that humans learn and apply effortlessly? We introduce LeCat, a two-part framework to analyze LLM's Learning of Category knowledge. First, we use LeCat to evaluate models on their ability to learn category knowledge, distinguishing between three cases: memorization of in-domain category knowledge, generalization to unseen instances, and ability to flexibly use knowledge in appropriate contexts. Then, we use LeCat to trace how this knowledge can be learned in pretraining from text that may encode category knowledge in various ways, such as verbatim repetition or exceptions, which we refer to as forms of "knowledge expression". We first assess open-source pretrained models using 199 curated category-feature pairs and analyze their training data. Our findings suggest that strengthening context and category-feature co-occurrence likelihood (PMI) improve learning for larger models and verbatim rule-like statements diminish flexible category learning. To systematically isolate these effects, we then train a 68M model from scratch with precise control over the categorical information each sees in their training data, confirming that verbatim repetition reduces flexible learning but mixture of diverse knowledge expressions lead to more robust category learning. Together, our findings highlight that while contextual flexibility is persistently difficult for models, how category knowledge is expressed in pretraining greatly impacts models' acquisition of robust category knowledge.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。涉及核心实体与技术议题包括:LECAT、预训练数据、LLM、类别知识,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Jimin Mun, Maarten Sap, Yanai Elazar),经 AITOP 多维评分管线结构化提炼。