LGMT 开源工具：测出 LLM 逻辑推理不一致率 20%-29%
LGMT: A Tool for Testing the Logical Reasoning Reliability of LLMs
一个测 LLM 逻辑一致性的开源工具，不需要标注答案就能跑，还带离线模式，做模型评测的可以试试。
研究人员发布了开源测试工具 LGMT，基于一阶逻辑的蜕变关系检测 LLM 推理不一致问题。工具内置 20 条蜕变关系，支持用 Z3 定理证明器验证自定义关系，可通过 CLI 或网页端运行，并提供无需 API key 的离线模式。对 6 个主流 LLM 在 3,091 组测试上的评测显示，模型在逻辑等价问题改写下的不一致率介于 20.84% 到 29.44% 之间。项目已在 GitHub 开源。
LGMT: A Tool for Testing the Logical Reasoning Reliability of LLMs
Large Language Models (LLMs) increasingly serve as reasoning components in modern software systems, yet a model that answers a logical query correctly can contradict itself when the same problem is restated in an equivalent form. Static benchmarks evaluate queries in isolation and therefore fail to expose such inconsistencies. To address this limitation, we present LGMT, an open-source tool that detects reasoning inconsistencies through metamorphic relations grounded in first-order logic. Given a source test case in premise-conclusion form, LGMT generates logically equivalent follow-up cases and checks whether the model's entailment predictions agree across the pair, without ground-truth labels. The tool provides 20 built-in metamorphic relations, supports custom ones verified by the Z3 theorem prover, and runs from a CLI or a web studio, including an offline mock mode requiring no API key. Across six widely used LLMs on a bundled benchmark of 3,091 metamorphic groups, LGMT reveals inconsistency rates ranging from 20.84% to 29.44%. LGMT is publicly available on GitHub at https://github.com/MaxwelsDonc/lgmt-tool, with a demonstration video at https://youtu.be/gou4opptu1U.