研究系统评估 LLM 对海地克里奥尔语的文化理解能力
Evaluating Cultural Awareness of LLMs for Haitian Creole
一篇关于小语种文化评估的论文，母语者出题测 LLM，发现海地角色老被写成苦难叙事，角度挺有意思。
arXiv 论文对 LLM 在海地克里奥尔语上的文化意识做了首次系统评估。研究团队由母语者整理出一组文化相关提示词，从具体性、偏见、多样性、变异性四个维度进行测试。结果显示克里奥尔语的文化意识明显落后于法语，且各领域表现不均衡，还受法语语言干扰影响。故事生成任务中，海地角色反复被刻画为苦难与坚韧的形象，说明正面描写也可能固化刻板叙事。
Evaluating Cultural Awareness of LLMs for Haitian Creole
Large language models (LLMs) exhibit substantial performance disparities between high- and low-resource languages. Beyond lower task performance, they often fail to capture the cultural norms and values of underrepresented communities. In this work, we present the first systematic evaluation of cultural awareness in LLMs for Haitian Creole, a language spoken by millions but severely underrepresented in digital resources. We assess cultural awareness along four complementary dimensions---specificity, bias, diversity, and variation---using a benchmark of culturally salient prompts curated by native speakers in a text infilling setting. Our results reveal a clear gap between cultural awareness in Haitian Creole and higher-resource French, with Haitian performance being more uneven across domains and more affected by French linguistic interference. Story generation further reveals recurring portrayals of Haitian characters through hardship and resilience, showing that even positive characterizations can encode stereotypical narratives. Our code, benchmark, and evaluation framework are publicly available.
