研究：MLLM 评审在文化分歧题上打分偏向单一文化
用 626 对中美图像题实测六个视觉模型当裁判时的文化偏向，连加示例都会加重偏差，做评测的人该看看
SFResearch 团队发布论文《Jury Duty》，测试视觉语言模型作为评审（MLLM-as-a-Judge）在文化模糊场景下的判断倾向。实验用 VOIR DIRE 基准，包含 626 对美国与中国大陆的图像题，覆盖食物、时尚和建筑。结果显示六个 MLLM 评审很少使用量表低端分数，导致争议题目向某一文化解读倾斜。Persona 提示只能部分纠正偏差，加入上下文示例反而加深倾斜。作者建议分别针对每个参照池单独报告对齐结果。
⚖️ Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity: https://t.co/iYleB4mlas
AI judges score the outputs that become training signal and leaderboard rankings. We tested whose judgment they reflect when cultures disagree.
→ VOIR DIRE: 626 paired U.S. and mainland Chinese image items across food, fashion, and architecture → Six MLLM judges rarely use the low end of the scale, which tilts contested items toward one cultural reading → Persona prompts only partly correct the tilt, and in-context examples deepen it
Our recommendation: report alignment against each reference pool separately.
#FutureOfAI #EnterpriseAI #EMNLP2026