FRAAC:面向法语象形图辅助通信的多模态大模型与数据集
FRAAC: A French Multimodal Large Language Model and Dataset forPictogram-Based Augmentative and Alternative Communication
研究团队开源了 30 万法语象形图平行语料,还训了一个能看象形图、也能用象形图回答的模型。做无障碍、辅助通信或多模态方向的人可以直接用这批数据和 RAPS 标注思路,做双语和多模态研究的也能借鉴。
法国研究团队发布 FRAAC,针对多模态大模型无法可靠解读和生成象形图序列的问题,提出 RAPS(Retrieval-Augmented Pictogram Selection)标注方法。该方法用 LLM 概念抽取和象形图词表检索,修正规则转录系统在命名实体、数字、多义词等场景下的失败案例,无需编写新规则。应用于法语后,RAPS 生成了 140 万平行语料对,其中 30 万已公开发布。
在人类排序研究中,标注员对同一句话的四种象形图转录进行排序,RAPS 超越了基于规则的最先进系统。研究还发现标准参考指标能区分质量差距明显的系统,但无法在强系统之间排出高低。团队最终构建了一个指令微调模型,支持多种 AAC 模态交互:接受象形图或文本输入,以象形图、语音或文本输出回答。
这项工作填补了辅助通信(AAC)领域文本-象形图平行数据稀缺和语法标准缺失的空白,为失语症等需要象形图交流的群体提供了可用的开源资源。
FRAAC: A French Multimodal Large Language Model and Dataset forPictogram-Based Augmentative and Alternative Communication
Pictograms are a core modality in augmentative and alternative communication (AAC), yet recent multimodal large language models (MLLMs) cannot reliably interpret or produce pictogram sequences. Two obstacles explain this gap: the absence of a standardized pictogram grammar and the scarcity of parallel text-pictogram data. We introduce Retrieval-Augmented Pictogram Selection (RAPS), an annotation method that refines a rule-based transcription system's output with LLM-based concept extraction and retrieval over a pictogram vocabulary, handling cases where hand-written rules fail (named entities, numbers, polysemous words) without requiring new rules. Applied to French, RAPS yields a synthetic corpus of 1.4 million parallel pairs, of which 300k are publicly released. In a human ranking study where annotators order four pictogram transcriptions of the same sentence, RAPS outperforms the rule-based state of the art. Moreover, we show that standard reference-based metrics separate systems of clearly different quality but fail to rank strong systems against each other. Finally, we build an instruction-tuned model that supports interaction in several AAC modalities: it accepts pictogram or text input and answers in pictograms, speech, or text.
本条情报归属 AITOP「论文」核心栏目,重点聚焦欧洲人工智能治理与技术生态。涉及核心实体与技术议题包括:FRAAC、RAPS、多模态、法语,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Alexandre Audibert, François Portet, Benoit Favre, Maja Jonck Hjuler),经 AITOP 多维评分管线结构化提炼。