论文

MemT-1.0 发布:把记忆内化为模型自身能力的新架构

MemT-1.0: Towards Memory-Native Evolution

精选理由

这篇论文提出了把记忆直接做进模型计算里的 MemT-1.0 架构,128K 长度下推理快 1.86 倍、存储省 66 倍,做智能体的可以看看记忆方案怎么做。

MemT-1.0 是一套记忆基础模型架构,基于 Metis 骨干,采用 Gated DeltaNet-2 实现轮次内稳定的 token 级记忆更新,并通过 Mixture-of-Storage 扩展原生记忆容量。训练采用继续预训练加后训练的两阶段流程,使用了 340 亿 token 的训练数据。评测显示 MemT-1.0-Nano 和 MemT-1.0-Flash 平均优于参数化记忆基线,与检索增强方法竞争力相当。在 128K 历史长度下,MemT-1.0-Nano 相比完整上下文 KV 缓存获得约 1.86 倍端到端推理加速和 66 倍持久存储缩减,同时提升了智能体任务的成功率。

原文 · arXiv cs.LG

MemT-1.0: Towards Memory-Native Evolution

Memory-native evolution requires making memory an intrinsic capability of foundation models, enabling them to continuously accumulate and utilize information. To this end, we introduce MemT-1.0, a new architecture of memory foundation models that internalize memory as part of model computation. Building on Metis, MemT-1.0 employs Gated DeltaNet-2 with a stable token-level memory update mechanism within each turn. We further introduce Mixture-of-Storage to scale native memory capacity. Following a deliberate backbone preparation, we develop a two-stage training pipeline consisting of continued pre-training and post-training, supported by 34B tokens of training data and efficient training and inference infrastructure. We release a model family comprising MemT-1.0-Nano and MemT-1.0-Flash, and conduct a comprehensive evaluation of memory and general capabilities. The results show that they outperform the parametric memory baselines on average and remain competitive with retrieval-augmented methods, while largely preserving general capabilities. At a history length of 128K tokens, MemT-1.0-Nano achieves approximately $1.86\times$ end-to-end inference speedup and a $66\times$ reduction in persistent storage relative to full context KV cache. By internalizing retrieved procedural experience, MemT-1.0 improves task success on agentic tasks, providing a practical step toward memory-native evolution.