THESEUS 框架评估小模型在知识图谱问答中的路径忠实度
The Path Matters: Evaluating Small Language Models Beyond Answer Accuracy in KGQA
研究者把知识图谱问答拆开看：小模型答案对了，走的路径未必对。用 PED 指标测出了 Hits@1 看不见的差距。
论文提出 THESEUS 框架，将知识图谱问答中的导航能力单独拆出来评估。冻结参数的小语言模型在每一跳从环境中给出的合法图操作中选择动作，决定是否停止，不涉及微调或自由生成答案。评估同时使用 Hits@1 和 Path Edit Distance（PED）衡量答案准确率与路径忠实度。在 Kinship 和 MQuAKE-ST 两个基准上，规模相近的模型在两个指标上表现分化，有时甚至指向不同模型，单一示例轨迹在不同模型上可能提升或降低导航效果。
The Path Matters: Evaluating Small Language Models Beyond Answer Accuracy in KGQA
Small language models (SLMs) are increasingly paired with knowledge graphs (KGs), yet end-to-end KG question answering conflates graph access, search, navigation, reasoning, and answer generation. This coupling makes it difficult both to determine whether an SLM can faithfully execute the reasoning path implied by a question and to attribute failures to navigation rather than to other stages of the pipeline. We isolate this capability by employing the THESEUS navigation and traceability framework and using frozen, off-the-shelf SLMs as local action policies. At each hop, the environment exposes the legal outgoing graph actions, and the model selects one executable graph action and decides whether to stop, without task-specific parameter updates, model-controlled beam search, or free-form answer generation. This controlled setting allows us to evaluate terminal-answer accuracy with Hits@1 together with path fidelity, using Path Edit Distance (PED) as the primary trajectory metric. Across the Kinship and MQuAKE-ST KGQAs, similarly sized local models differ substantially in answer accuracy and path fidelity, with the two metrics sometimes favoring different models. This model-dependent behavior also extends to prompting, as a single demonstrated trajectory can improve or degrade navigation depending on the model. These results motivate evaluating SLM graph reasoning beyond endpoint accuracy alone.
