MAPLE 框架拆解 LLM 知识提取增强技术的真实效果
MAPLE: A Multi-Axis Parametric Knowledge Extraction Lens for LLM Evaluation
这篇论文搞了个叫 MAPLE 的评估框架，把六种 KEE 技术在 OLMo2、Qwen2.5、Llama3 上逐层拆开看，比只看准确率细多了。
论文提出 MAPLE 框架，用于评估激活转向、解码调整、提示词等知识提取增强（KEE）技术对 LLM 参数化知识提取的影响。传统评估只用总体准确率，会把修复的失败和破坏的成功混在一个分数里。MAPLE 在表层、概率层和超出采样层三个维度分别测量答案恢复与干扰、失败案例中 gold-token 排名变化、固定预算采样漏掉答案的找回率。研究在 OLMo2、Qwen2.5、Llama3 三个模型家族上评测了六种 KEE 技术，发现不同技术存在明显不同的效果与权衡。
MAPLE: A Multi-Axis Parametric Knowledge Extraction Lens for LLM Evaluation
LLMs absorb vast knowledge from training data, yet their inherently probabilistic nature offers no guarantee that this knowledge will surface in their outputs. To bridge this gap, Knowledge Extraction Enhancement (KEE) techniques such as activation steering, decoding adjustments, and prompting modify inference, potentially reinforcing existing predictions or surfacing less accessible answers. Yet their evaluations centered on aggregate accuracy, which obscures diverse effects by collapsing recovered failures and disrupted successes into a single score. We introduce MAPLE, a unified framework for characterizing how KEE techniques affect parametric knowledge extraction. Its diagnostic metrics disentangle these effects at the surface, probability, and beyond-sampling levels by measuring answer recovery and disruption, gold-token rank shifts on unrecovered failures, and recovery of answers missed by fixed-budget sampling. We evaluate six KEE techniques spanning decoding, activation steering, and prompting across three model families: OLMo2, Qwen2.5, and Llama3. Our analysis reveals distinct effects and trade-offs across techniques, highlighting the value of MAPLE for understanding and comparing KEE beyond aggregate accuracy.
