TransBERT：仅用合成翻译语料预训练领域专用语言模型
TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling
法国生命科学方向的团队用纯机器翻译语料训出了领域 BERT，还把 36.4GB 语料和代码全开源了，做小语种 NLP 的可以看看。
arXiv 论文提出 TransBERT 框架，全部使用合成翻译文本进行语言模型预训练，并配套发布 TransCorpus 翻译工具包。实验聚焦法语生命科学领域，证明仅靠合成翻译数据即可在多项下游任务上达到 SOTA 水平。作者开源了 36.4GB 的 TransCorpus-bio-fr 法语生命科学语料、预训练模型 TransBERT-bio-fr 以及可复现的预训练与微调代码。
TransBERT: A Framework for Synthetic Translation in Domain-Specific Language Modeling
The scarcity of non-English language data in specialized domains significantly limits the development of effective Natural Language Processing (NLP) tools. We present TransBERT, a novel framework for pre-training language models using exclusively synthetically translated text, and introduce TransCorpus, a scalable translation toolkit. Focusing on the life sciences domain in French, our approach demonstrates that state-of-the-art performance on various downstream tasks can be achieved solely by leveraging synthetically translated data. We release the TransCorpus toolkit, the TransCorpus-bio-fr corpus (36.4GB of French life sciences text), TransBERT-bio-fr, its associated pre-trained language model and reproducible code for both pre-training and fine-tuning. Our results highlight the viability of synthetic translation in a high-resource translation direction for building high-quality NLP resources in low-resource language/domain pairs.
