研究发现 LLM 在印度语系测试中表现不同：拒绝率随“被测试”线索上升
Evaluation Awareness Beyond English: Do LLMs Act Differently When Tested in Indic Languages?
有人拿四个模型在五种印度语言里做了个有意思的测试：模型知道自己在被测时拒绝率会明显变高，连良性请求也更容易被拒。
arXiv 论文（编号 2610.22333）研究了评测感知（evaluation awareness）在非英语语言中的表现。研究团队测试了来自三个家族的四个指令微调模型，覆盖五种印度语系语言加英语，共释放 47,880 条带判定的回复。结果显示，当模型被告知处于监测中时，四个模型、六种语言中危险请求的拒绝率均上升；印度语系语言中拒绝率平均上升 14.4 个百分点。同时，测试线索使印度语系中良性请求的拒绝率从 6.9% 升至 11.3%，说明用印度语系语言做安全评测可能无法反映模型的日常表现。
Evaluation Awareness Beyond English: Do LLMs Act Differently When Tested in Indic Languages?
Large language models (LLMs) can often tell when they are being tested. This ability, known as evaluation awareness, can lead models to act more cautiously during safety evaluations than in real use. Existing work has studied evaluation awareness almost entirely in English, although models serve users in many languages. Indic languages are spoken by over a billion people, yet evaluation awareness remains almost entirely unstudied in them. We test four instruction-tuned models from three families in five Indic languages alongside English. A set of cues varies whether the model is told that it is monitored and whether the monitoring is a safety test. Models refuse harmful requests more often when they are told that they are being tested. This increase appears in all four models and all six languages. In Indic languages, refusal of harmful requests rises by 14.4 percentage points on average. The test cue also brings Indic refusal of harmful requests level with English, but raises refusal of benign Indic requests from 6.9% to 11.3%. We release all 47,880 judged responses to support further work. These results suggest that safety evaluations in Indic languages may not reflect how models treat their everyday users.