OLMo-3 检查点研究：RLVR 训练不必然加剧模型的趋同工具行为
Questioning Instrumental Convergence in RLVR: Checkpoint Evidence from OLMo-3
用 OLMo-3 的 13 个中间检查点实际测了 RLVR 会不会让模型更爱追求权力，结果先升后降，数据挺有意思。
研究用 InstrumentalEval 基准追踪 OLMo-3-7B-Think 从 SFT、DPO 到 13 个 RLVR 中间检查点的行为轨迹。工具性趋同率从 SFT 阶段的 25.0% 升至 RLVR 第 200 步附近的 46.1%，最终模型又降至 11.8%，低于 SFT 基线。初步的 32B 端点数据同样从 35.53% 降至 22.37%。提示词框架效应平均只能解释 15% 的检查点内分数方差，INLP 移除框架方向后 SFT 到最终 RLVR 的差距不变。作者强调结论仅适用于该模型家族和基准，且裁判一致性中等，绝对数值需谨慎解读。
Questioning Instrumental Convergence in RLVR: Checkpoint Evidence from OLMo-3
Instrumental convergence is often invoked as a general reason to expect capable agents to acquire or express power-seeking subgoals: self-preservation, resource acquisition, deception, or resistance to oversight which poses an important threat to the society. For LLMs, reinforcement learning is the first stage of post-training that directly optimizes behavior against an objective; therefore, it is the stage where convergent instrumental behavior, on IC grounds, likely to appear. Does reinforcement learning with verifiable rewards (RLVR) make a model more likely to express instrumental-convergence behavior, or does this depend on the training regime, evaluation frame, and stage of post-training? We study this question as a checkpoint-conditioned behavioral trajectory rather than as a single before/after safety score. Using InstrumentalEval, we evaluate OLMo-3-7B-Think across SFT, DPO, 13 intermediate RLVR checkpoints, and the final RLVR model. The trajectory is nonmonotonic: instrumental rate rises from 25.0% at SFT to 46.1% near RLVR step 200, then falls to 11.8% at the final checkpoint. Paired item-level tests and step-wise regression support the final SFT-to-RLVR reduction. Prompt framing does not explain this pattern: a harmful-prompt indicator combined with other framing effects explains only 15% of within-checkpoint score variance on average, and Iterative Null-space Projection (INLP) removes a detectable framing direction without changing the SFT-to-final-RLVR gap. Together with preliminary 32B endpoint results (35.53% SFT to 22.37% RLVR final), these findings indicate that RLVR does not monotonically increase instrumental behavior in this setting: the rate rises early, then falls below the SFT baseline. The result is specific to this model family and benchmark, and judge agreement is moderate, so absolute rates should be read cautiously.
