Ru-Jev 307M 交叉编码器实验:标签改写增强对陌生词汇有效,缩写无效
Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Words, Not on Abbreviations or Codes
做意图路由或分类的朋友可以看看:同一模型换个标签写法,准确率能差这么多,论文还给了四个评测坑点。
一篇 arXiv 论文对比了 307M 参数多语言交叉编码器 Ru-Jev 的两个版本,新版在 13 个俄语任务训练数据上加入了标签多种表面形式的增强。在 18 个路由标签、10 套词表、每题 5 或 18 个候选的评测中,新版对普通词和外来语标签准确率最高提升 0.24,14 项增益中 11 项通过 Holm 校正。但缩写标签没有提升,去除与训练选项相关的标签后,两个版本在不透明编码上都处于或低于随机水平。在两个未见过的新数据集上,准确率分别变化 -8.4 和 +5.6 个百分点。
Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Words, Not on Abbreviations or Codes
Typed-decision models score options that the caller supplies at request time, so their accuracy depends on label wording that the model's developer does not control. We compare two releases of a 307M-parameter multilingual cross-encoder, Ru-Jev, trained on thirteen Russian-language tasks; the later release adds an augmentation that presents labels in several surface forms. Holding questions fixed and varying only how eighteen routing labels are worded, we evaluate ten label vocabularies, produced by three separate processes and checked for overlap with the training data, with five candidates per question and, for eight of them, also with eighteen. On held-out vocabularies made of ordinary or borrowed words (English-derived jargon, office slang, department names), the augmented release is more accurate in every setting, by up to 0.24. Eleven of these fourteen gains survive a Holm correction (slang with eighteen candidates and descriptive sentences with five do not), and for jargon on all labels the label-level interval excludes zero in both settings. On abbreviations and opaque codes it gives no usable gain: on abbreviations it is not better, and worse on the clean slice, and on opaque codes both releases stay at or below chance once labels related to training options are removed. In most cells where accuracy rises, the risk-coverage area falls, but with eighteen candidates confident errors rise on jargon and slang. On two datasets neither release saw, accuracy moves by -8.4 and +5.6 points. We also report four evaluation hazards from comparing the two releases.