SKIM 方法:把对话记忆直接写进冻结语言模型参数
Self-Keyed In-Parameter Memorization for External Memory-Free Dialogue Recall in Frozen Language Models
不用向量库和外部存储,论文让模型把对话记在自己参数里,在 5 个模型上验证有效,做长对话记忆的可以看看。
arXiv 论文提出 self-keyed in-parameter memorization(SKIM),在无外部存储的假设下研究对话记忆问题。SKIM 让模型对每轮对话自动生成多个问题,并把这些问题作为键写入模型参数,后续用户提问可匹配其中一个完成召回。该方法在 5 个语言模型上评估,能从参数中召回大部分已存储对话轮次,全程不依赖外部存储。
Self-Keyed In-Parameter Memorization for External Memory-Free Dialogue Recall in Frozen Language Models
A large language model maintains conversational content in an external storage that is outside the model and grows as the dialogue continues. This work assumes an environment where a language model has no external memory and no prior knowledge of the future question or its answer. The motivation is to research whether a dialogue can be stored in the model parameters and later answer a question about it. This work proposes self-keyed in-parameter memorization (SKIM), a closed-form method in which the model generates its own questions about each dialogue turn and writes those questions into its parameters as the keys that later retrieve the turn's content. Generating many varied questions for each turn allows a later user question to match one even when phrased differently, and its answer is recalled from the parameters alone. Evaluated on five language models, SKIM recalls most stored dialogue turns from later user questions, and it answers from the model parameters without any external storage.