arXiv 论文提出 isloss：检测 LLM API 转换中对话信息丢失的确定性测试方法
Testing Conversation Preservation Across LLM APIs: A Rule-Based Oracle for Detecting Silent Information Loss
做 LLM 网关、多模型切换或者翻译层的开发者该看看这篇：翻译后的请求能过 API 校验，但角色、工具调用信息可能已经悄悄丢了。论文给出的 ConvIR 中间表示加规则判定，不用额外调用模型就能测出问题，还附带真实故障率数据。
arXiv 论文 2611.15543v1 提出 isloss，一个用于测试 LLM API 对话信息保留的确定性 oracle。它将源请求与翻译后的请求、回复映射到共享中间表示 ConvIR，再用显式规则区分合法的表示变化与信息丢失，无需参考翻译，也无需额外模型调用。
评估采用基于规范的故障注入、人工审查以及对翻译器、网关和客户端库的实验：在留出测试集上检测出 125 个植入故障中的 95 个（76.0%），误报为 100 次中的 2 次（2.0%）。对五个翻译器代码库六种配置的合成请求套件测试发现，685 个完成翻译中有 127 个（18.5%）通过了目标 API 的静态校验，却在静默改变 prompt、角色或工具信息，而这些信息目标 API 本可保留。回复翻译中，558 个流里有 203 个（36.4%）在至少一个官方 OpenAI SDK 中引发异常或静默损坏。
针对六个模型的实验表明，当任务依赖受影响信息时，翻译引起的变化会改变模型行为。论文将对话信息保留确立为 LLM 互操作、智能体执行与模型评估之外的一项独立正确性要求。
Testing Conversation Preservation Across LLM APIs: A Rule-Based Oracle for Detecting Silent Information Loss
LLM applications increasingly rely on translators and compatibility layers to move conversations across APIs. Yet a translated request can be accepted by the target API while silently losing information about instruction roles, tool interactions, error states, or generation controls. We introduce isloss, a deterministic oracle for testing conversation-information preservation across LLM API translations. It maps source and translated requests and replies to a shared intermediate representation, ConvIR, and applies explicit rules to distinguish legitimate representation changes from information loss. Unlike schema validation or reference-based differential testing, isloss requires neither a reference translation nor additional model calls. It also distinguishes avoidable loss from degradation caused by target API limitations. We evaluate isloss using specification-based fault injection, human review, and experiments on translators, gateways, and client libraries. On held-out tests, it detects 95/125 planted faults (76.0%) with 2/100 false alarms (2.0%). Across a synthetic request suite spanning six configurations of five translator codebases, 127/685 completed translations (18.5%) pass static target checks while silently changing prompt, role, or tool information in ways the target API could have preserved. In translated replies, 203/558 streams (36.4%) cause exceptions or silent corruption in at least one tested official OpenAI SDK. Targeted experiments across six models show that translation-induced changes can alter model behaviour when tasks depend on the affected information. These results establish conversation preservation as a distinct correctness requirement for reliable LLM interoperability, agent execution, and model evaluation.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：LLM API、isloss、ConvIR、OpenAI SDK，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始信息发布自一手权威信源「arXiv: OpenAI」（Tier-1 一手发布），具备高确定性参考价值。截至目前，全网已有 11 家独立信源对该事件进行交叉印证与跟进报道（含 OpenRouter、TestingCatalog、IT之家 等），属于近期高关注度行业信号。
