研究：LLM-as-a-Judge 提示策略效果依赖评审模型与语言
Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness
用 27 种语言和超百万条评分实测发现：CoT 这类提示技巧换个评审模型可能就失效，做多语言评测的得看看。
arXiv 论文构建了一个包含 9 种提示策略、4 个开源评审模型、27 种语言和 4 类任务的基准，共产生超过 100 万条逐点评分，并经人工判断验证。结果显示 Chain-of-Thought 等提示策略无法跨模型迁移：评审模型的选择比提示策略的选择影响更大，同一策略在不同评审模型上甚至效果方向相反。低资源语言下稳定性明显下降，而汇总式报告会掩盖这些差异。论文结论是提示策略需针对具体评审模型、任务和语言逐一验证。
Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness
LLM-as-a-Judge evaluation is increasingly applied to multilingual tasks, yet prompting strategies such as Chain-of-Thought and Role Conditioning are recommended based solely on English benchmarks. Whether their effectiveness transfers across judge models and languages remains unknown: prompt-sensitivity studies are largely English-centric, while existing multilingual judge studies fix the prompt template. In this work, we address this gap by measuring this interaction through a benchmark of 9 prompting strategies, 4 open-weight judges, 27 languages, and 4 tasks across 13 datasets, yielding over 1M pointwise scores on gold-reference outputs, validated against human judgments. We find that prompting strategies are not transferable: the choice of the judge model dominates the choice of prompting strategy, strategy effects reverse direction across judges, and stability degrades in low-resource languages, where these effects are concealed by aggregate reporting. These findings show that prompting effects in multilingual evaluation are conditioned on the judge, task, and language. Consequently, prompting strategies should be validated for the intended evaluation setting, and cross-study comparisons should account for both judge and prompt configuration.
