Context–Answer Metamodels：在首个答案 token 生成前预测 LLM 激活
Context--Answer Metamodels: Forecasting LLM Activations Before the First Answer Token
用读上下文的激活就能预测模型答案阶段的激活，跨 Qwen2.5 到 Gemma 4 都能迁移，做可解释性或行为预测的可以看看这篇。
一篇 arXiv 论文提出 context–answer metamodels，把模型读上下文时的激活序列映射为答案激活的点预测或分布预测。实验覆盖 4 种 metamodel 家族，训练集规模扩展到 500,000 条 prompts 时预测精度持续提升，且能零样本泛化到域外数据集。预测结果保留了行为层面的预测信号。Qwen2.5 的上下文激活可以预测 Qwen3.5 和 Gemma 4 的答案激活，说明激活预测能跨模型迁移。
Context--Answer Metamodels: Forecasting LLM Activations Before the First Answer Token
Under stochastic decoding, a single context can elicit many different answers from a language model. Characterizing this variation typically requires many expensive autoregressive rollouts, each revealing only one realization. We ask whether the activations produced while reading the context can directly forecast the model's rollout activations. We train context--answer metamodels that map a context activation sequence to point or distribution forecasts of mean-pooled answer activations. We find that (i) the four metamodel families we study forecast mean-pooled answer activations with high fidelity, and the distribution metamodels also capture rollout variability; (ii) performance improves as the training set grows to 500,000 prompts, and the metamodels generalize zero-shot to out-of-domain datasets; (iii) their forecasts retain behavior-predictive signal; and (iv) context activations from Qwen2.5 predict mean-pooled answer activations from Qwen3.5 and Gemma 4, suggesting that activation forecasts can transfer across models. Together, these results suggest that context activation traces can support scalable forecasts of behaviors in LLM answers before the first answer token is generated.
