论文提出双通道约束 GRPO，让 VLM 安全训练像预算一样可控
Safety as a Budget: Calibrating the Control Channel of Constrained GRPO for Vision-Language Models
做 VLM 安全 RL 的人可以直接抄这套方法：不用反复调奖励权重，把安全率当预算设定就行。论文里两个失效模式（打分器均值有偏、组内标准化吃掉惩罚）解释得很清楚，Qwen3-VL-2B 上的数据也扎实。
一篇 arXiv 论文提出双通道约束 GRPO（two-channel constrained GRPO），把 VLM 的安全训练从调奖励权重改成显式设定预算：有害协助率和过度拒答率两个约束由 Lagrange 乘子直接控制。作者指出朴素做法有两处失效——为排序设计的打分器均值有偏，导致乘子收敛状态由偏差而非预算决定；标准 GRPO 组内标准化会把乘子的惩罚项抵消掉。修复方法是基础奖励保留原始分数做组内比较，惩罚与约束估计经单调映射校准后独立进入优势函数。
实验在 Qwen3-VL-2B 上进行，用开源 4B 模型的免训练读出作打分器：1037 条对抗提示上的 held-out 有害率从标准 GRPO 的 8.9% 降到 4.8%（未校准预算为 7.6%），全部随机种子都更低；400 条良性查询的过度拒答不超过 2%，MMBench 成绩不变。意图藏在图像中的样本改善最大（10.1% 到 5.1%），在未见过的 HADES 基准上从 28.9% 降到 12.8%。固定点分析预测的控制器状态在每个种子下都被保序分数扰动复现，结论还通过另一评审模型的映射、图像评审和人工审计验证。
Safety as a Budget: Calibrating the Control Channel of Constrained GRPO for Vision-Language Models
Reinforcement learning can teach vision-language models (VLMs) to decline harmful image-text requests, but how safe the result becomes is set indirectly, through reward weights found by trial runs. Stating safety as budgets on the rate of harmful assistance and the rate of over-refusal, enforced by Lagrange multipliers inside GRPO with an off-the-shelf scorer as the cost, is the natural fix, and it fails twice. The multiplier compares the scorer's mean with the budget, and scorers built for ranking get the mean wrong: a fixed-point analysis shows that the mean bias, not the budget, then decides whether the multiplier settles, saturates or switches off. And standard GRPO standardizes the penalized reward within each group, which cancels the multiplier before it reaches the gradient. Two-channel constrained GRPO repairs both: the base reward keeps the raw score for the within-group comparison, the penalty and the constraint estimate pass through a monotone map fitted on judge-labeled training responses, and the advantage keeps the multiplier. On a Qwen3-VL-2B policy with a training-free readout of an open 4B model as the scorer, held-out harm on 1,037 adversarial prompts falls from 8.9% under standard GRPO to 4.8%, lower in every seed, where the uncalibrated budget gives 7.6%; over-refusal stays at most 2% on 400 benign queries and MMBench is unchanged. The gain is largest where the intent sits in the image (10.1% -> 5.1%) and grows on the unseen HADES benchmark (28.9% -> 12.8%). Order-preserving distortions of the scores reproduce each predicted controller state in every seed, and the results hold under a map fitted on another judge's labels, an image-reading judge and a human audit.
本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：GRPO、Qwen3-VL、VLM、AI安全，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.AI」（@Bo Han），经 AITOP 多维评分管线结构化提炼。
