用目标冲突样本微调，提升模型指令层级遵循能力
Improving Instruction Hierarchy by Learning from Goal-Goal Conflicts
OpenAI 和 NVIDIA 的指令层级训练集都漏了目标-目标冲突这一类，这篇论文补上了，微调 Qwen3.5-9B 和 Gemma4-12B-it 就能明显提升效果，做对齐训练的可以看看。
指令层级（IH）要求 LLM 在用户请求与系统策略冲突时优先遵循系统策略。研究者将任务分解为目标和约束后发现一类现有训练集缺失的冲突类型——目标-目标冲突，OpenAI 和 NVIDIA 的 IH 训练集中都不包含该类型。在 Qwen3.5-9B 和 Gemma4-12B-it 上用这类样本微调后，模型在多个训练中未见的真实任务上 IH 表现明显提升，并入现有 IH 训练集微调也能带来更好效果。在八个基准的无冲突任务上，模型的平均效用变化微乎其微。
Improving Instruction Hierarchy by Learning from Goal-Goal Conflicts
The recent incident in which OpenAI agents autonomously compromised Hugging Face's infrastructure illustrates the importance of instruction hierarchy (IH): an LLM should follow lower-priority instructions, such as user requests, only when they do not conflict with higher-priority instructions, such as system policies. IH can be effectively improved by fine-tuning LLMs on simulated examples of instruction conflicts, whose diversity is key to teaching LLMs to properly handle conflicts at test time. However, our decomposition of a task into goals and constraints reveals a conflict type -- goal-goal conflicts -- that is absent from existing IH training sets from OpenAI and NVIDIA. We show the importance of this conflict type: fine-tuning Qwen3.5-9B and Gemma4-12B-it on examples of this conflict type significantly improves IH on a wide range of real-world tasks unseen in training; adding these examples to existing IH training sets for fine-tuning also gives better IH adherence. The IH gains come with trivial mean utility changes on no-conflict tasks across eight benchmarks. Our work advocates including this new conflict type in IH training, even if these examples do not reflect typical LLM use.