ConfabQA 基准：用隐藏状态线性探针预测小模型幻觉还是拒答
Geometry of Confabulation and Refusal in Small Language Models
有人拿 Qwen3、Gemma 2 这些小模型研究怎么提前判断幻觉，还能直接翻转成拒答，代码都开源了，玩 SAE 的别错过。
arXiv 论文 2611.01272 提出 ConfabQA 基准，包含 784 个条目，用于区分小语言模型面对无答案问题时会拒答还是编造答案。作者在 Qwen3-1.7B 和 Gemma 2 2B 上用线性探针、稀疏自编码器（SAE）和干预方向优化分析隐藏状态，探针相对提示特征基线分别提升 7.4 和 16.3 个百分点。在 Llama 3.2 3B 上，正确性信号可在 ConfabQA、PopQA 和 TriviaQA 三个基准间迁移，保留 81% 的探针边际。沿拒绝向量方向对 Qwen3-1.7B 隐藏状态施加低于自身范数的推动，可将多数编造回答翻转为以 As 开头的拒答开场。
Geometry of Confabulation and Refusal in Small Language Models
Faced with an unanswerable question, small language models can either refuse or confabulate: confidently produce a wrong answer. Can the hidden state before the first generated token provide a clue? Can we flip confabulations into refusals? I introduce ConfabQA, a 784-item benchmark whose obscure-pre-cutoff category breaks the typical correctness/knowledge-cutoff confound and map the geometry of the hidden state with linear probes, a sparse autoencoder (SAE), and direct optimization of intervention directions. On Qwen3-1.7B and Gemma 2 2B the probe separates upcoming refusals from confabulations well beyond prompt features (+7.4/+16.3 pp); on Llama 3.2 3B the ConfabQA margin is within fold-to-fold spread, but a direct refusal probe on PopQA and TriviaQA recovers +18-25 pp. On Qwen I closely inspect three candidate directions for the refusal vector: probe-recovered, Qwen-Scope SAE feature, and directly optimized. Although the three directions are pairwise 50-81 degrees apart, a push at or below the state's own norm along any of them flips most confabulations into refusal openings dominated by the "As..." family. Forcing the opener without touching the hidden state yields the same one-in-three judged-refusal rate as any push. The opposite push collapses refusals mostly into confabulations, yet leaves originally correct answers intact, so the direction gates the opening speech act rather than the knowledge behind it. Confabulations do not resolve into a concentrated feature in a dictionary or a token signature on Qwen3-1.7B. Scoring every probe on the margin over the strongest of four prompt-feature baselines extracts the hidden state's own contribution. Correctness signal is model-dependent and portable only for Llama 3.2 3B: 81% of the probe's margin over test majority survives transfer between ConfabQA, PopQA, and TriviaQA. Benchmark, judge, and code are released.