Stanford 论文揭示小模型少样本学习的内部电路
Stanford 拆解了 Qwen3 和 Llama 小模型的少样本学习电路，还给了一条实用建议：写 few-shot 提示时把每个标签多重复几次。
Stanford 论文在 Qwen3-4B、SmolLM3-3B、Llama-3.2-3B 上追踪了少样本学习的完整路径。LLM 的上下文学习依赖对已存储标签的查找，标签首次重复时才触发这一机制。每类 1 个样本时模型正确率约 20%，增加到 6 个样本时升至 96%。约 90 个后续注意力头将新句子与存储的类别匹配。因此示例无法教会模型本身不存在的特征，比如单词的第三个字母。
New Stanford paper finds that showing a small AI language model more examples cannot teach it a pattern it does not already notice, such as the third letter of a word.
finds that LLMs learn from few-shot examples by looking up stored labels, and only switch that lookup on when labels repeat.
An early signal opens this lookup only when some label repeats. With 1 example per class the model was right about 20% of the time, and with 6 it was right 96%.
Give a model a dozen sentences tagged with made-up labels like foo and bar, and it learns the mapping without weight updates. Earlier work found parts of this, but not how they fit together.
They traced the full path in Qwen3-4B, SmolLM3-3B and Llama-3.2-3B. Each label token stores the class of its sentence, and about 90 later heads match the new sentence against those stored classes.
For few-shot prompts, repeat every label several times and pick classes the model can already tell apart, since examples cannot teach a feature it lacks.
– arxiv. org/abs/2611.05344
Title: "A Circuit for In-Context Learning"
