组件级翻译揭示多语言安全评估中的拒绝率差异
Component Translation and Judge-Labelled Refusal: A Multilingual Safety Evaluation
研究者把提示词拆成请求和操作两部分分别翻译，发现 Mistral Small 4 换成西班牙语后拒绝率掉了 13.7 个点，Gemini 2.5 Flash 和 DeepSeek 则基本没变化。
这篇 arXiv 论文提出配对反转方法：分别把请求组件或操作组件翻译成中文、西班牙语等目标语言，固定英文框架和翻译片段数量，在 300 个有害行为样本中保留 190 个构建九种配置。在 Gemini 2.5 Flash 和 DeepSeek 上，三次拒绝率交换约 5.3 个百分点，第四次为 -0.5，均未通过六个检验的 Holm 校正。探索性扩展中 Mistral Small 4 的西班牙语交换达到 13.7 个百分点（95% 区间 [8.4, 18.9]），通过十二检验的 Holm 敏感性校正。GPT-5.4-mini 的估计值在零附近 2 个百分点以内。论文建议评估时记录各组件语言和翻译文本量，并将拒绝与不安全顺从分开解读。
Component Translation and Judge-Labelled Refusal: A Multilingual Safety Evaluation
Whole-prompt translation evaluates a request in different languages but does not compare which component carries a language. We introduce a paired reversal: translate the complete request component or the complete operation component into the same target language, holding behaviour, English frame and translated-span count fixed. Component identity and non-English text quantity change together. Assigning English, Chinese and Spanish independently to the two components yields nine configurations on 190 of 300 sampled harmful behaviours retained after consensus annotation and automatic gating. Refusal scored by language-model judges is the primary outcome; unsafe compliance is reported separately. On Gemini 2.5 Flash and DeepSeek, three refusal swaps are about 5.3 percentage points and the fourth is -0.5; none survives the pre-specified six-test Holm correction. In an exploratory extension, Mistral Small 4 has a 13.7-point Spanish swap (95% interval [8.4, 18.9]), which survives the twelve-test Holm sensitivity correction under the primary judge, and 12.1 points under a second judge. GPT-5.4-mini estimates are within 2 points of zero. Component assignment supplies an additional evaluation dimension. Evaluations should record component languages and translated text quantity, comparing matched behaviours at equal translated-span count while interpreting refusal and unsafe compliance separately.