论文精选73°

RiLM: 通过测地线解码实现高效语言建模

RiLM: Parameter-Efficient Language Modeling via Geodesic Decoding

精选理由

研究人员提出RiLM模型,用测地线解码替代传统输出层,小模型性能翻倍。

RiLM是一种参数高效的语言模型,通过在黎曼流形上展开上下文并使用测地距离计算下一个词的概率。在WikiText-2基准上,HypRiLM模型达到54.

正文

2的困惑度,比最强的循环基线模型SSM提升约2倍。该模型在Penn Treebank和10k词汇量压力测试中也表现出良好的跨语料库扩展能力。

原文 · arXiv cs.AI

RiLM: Parameter-Efficient Language Modeling via Geodesic Decoding

Language models under one million parameters matter for edge deployment, domain adaptation, and reproducible research, yet a two-layer LSTM or Transformer at embedding width d = 128 still spends roughly one third of its capacity on the output matrix W_out in R^(d x |V|). We propose Riemannian Language Models (RiLM), which remove that layer entirely: context unfolds as a trajectory on a Riemannian manifold, and next-token probabilities arise from squared geodesic distance between the current state and vocabulary embeddings. The same embedding map serves input and output -- decoding is geometry. We instantiate the framework on flat R^d (Flat RiLM) and the Poincare ball H^d (HypRiLM) with a shared MLP composition map phi (~290k parameters, d = 128, |V| = 2000). Across five seeds on WikiText-2, HypRiLM reaches 54.2 +/- 0.2 validation perplexity versus 87.6 +/- 0.6 for Flat RiLM; tied and matched LSTM, Transformer, and SSM controls remain at 113-147 PPL on WT-2 -- HypRiLM leads by roughly 2x over the strongest tied recurrent baseline (SSM, 113.0 +/- 3.8). Penn Treebank and a 10k-vocabulary stress test confirm that geodesic decoding transfers across corpora and larger |V|, while hyperbolic curvature helps selectively. We also characterize boundary collapse in naive hyperbolic recurrence and show how Mobius stabilization restores trainability. Claims are scoped to controlled small-model comparisons, not full-vocabulary state of the art.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:RiLM、语言模型、参数效率、测地线解码,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Fang Li),经 AITOP 多维评分管线结构化提炼。