G-Retriever 软提示的 token collapse 现象分析
On the Token Collapse Phenomenon in Textual Graph Soft Prompting
用图结构增强 LLM 提示的朋友看看这篇：G-Retriever 的图 token 其实塌缩成常量了，作者还给了检测方法和修复流程，代码开源。
arXiv 论文分析了 G-Retriever 的图软提示机制，发现一种称为 token collapse 的病态行为：在 3 个基准、3 种图编码器、2 个 LLM 家族上，图 token 几乎不随输入变化。实验显示，把 token 换成数据集均值或来自损坏图的 token，准确率都不受影响，而一个从不读取图的固定常量提示在全部 3 个基准上追平或超过 G-Retriever。作者提出两阶段训练流程可生成非塌缩 token，并给出干预与解码探针方法来评估 token 是否真正携带图信息。
On the Token Collapse Phenomenon in Textual Graph Soft Prompting
Recent methods for injecting graph structure into LLM prompts include graph soft prompting, in which a graph neural network (GNN) learns graph-specific tokens that capture relational information among entities in textual prompts. G-Retriever combines graph soft prompts with a verbalized description of the same graph to strengthen the textual prompt signal. We conduct an in-depth analysis of G-Retriever and uncover a pathological behavior we term token collapse. Across three benchmarks, three graph encoders, two LLM families, and both frozen and LoRA-adapted models, G-Retriever's graph tokens become almost indistinguishable across inputs. We provide evidence that the same failure is inherited by subsequent methods, even with multi-token pooling. Under the standard setup, swapping a token with the dataset mean, a token from another example, or one derived from a corrupted graph produces no resolved answer-level effect in accuracy. Moreover, a learned constant prompt that never accesses the graph matches or surpasses G-Retriever on all three benchmarks. These findings indicate that, rather than conveying graph content as intended, the collapsed token functions primarily as a generic soft prompt. We also examine conditions linked to this collapse and propose a two-stage procedure that yields non-collapsed tokens while maintaining competitive performance in most cases. However, token variability alone does not prove that the model uses graph information: via interventions and decoding probes, we show that depending on the configuration, a token can reflect the question, an answer prior, or graph-derived content that the LLM may or may not use. Overall, our results offer a practical approach to evaluating graph-token alignment rather than inferring it from task accuracy alone. Code to reproduce our experiments is available at https://github.com/difra100/Token_Collapse_KGQA.
