PFR 测试揭示：类型化决策模型的 Fallback 机制反而引入错误
Lost in the Fallback: How Reliably Do Typed Decision Models Use Fallback Options?
做请求路由或模型选择的可以看看：给模型留的兜底选项实测常出错，规则换个说法决策就变，这篇把坑测得很细。
一篇 arXiv 论文提出 Paired Fallback Robustness（PFR）测试，检验类型化决策模型（TDM）能否可靠使用 Fallback 选项。测试考察两个维度：模型能否在常规类别与 Fallback 之间正确切换，以及面对逻辑等价的同一规则表述时决策是否一致。作者在 3 个基准上评测开源决策模型，覆盖不同架构与参数规模的对照。结果显示，正确识别类别并不等于正确触发 Fallback，即使正确类别被明确给出，错误依然出现；逻辑等价的规则表述也会让决策出现明显分歧。论文指出，本用于提升可靠性的 Fallback 机制反而可能成为错误来源。
Lost in the Fallback: How Reliably Do Typed Decision Models Use Fallback Options?
Typed decision models (TDMs) are increasingly used in applications such as request routing and model selection, making reliable decisions critical to system performance. Fallback options accommodate categories outside regular choices, enabling deferral to human experts or more capable models. While recent works have examined adversarial vulnerabilities in TDMs, their ability to follow fallback rules in non-adversarial settings remains underexplored. We introduce the Paired Fallback Robustness (PFR) Test to evaluate two aspects of fallback reliability: whether models correctly switch between regular and fallback options, and whether they make consistent decisions under logically equivalent definitions of the same rule. We evaluate open-source decision models on three benchmarks, with controlled comparisons across architectures and parameter counts. We find that correct recognition does not reliably transfer to correct fallback options, and errors persist even when the correct category is explicitly supplied.Moreover, logically equivalent formulations of the same fallback rule can also produce substantial disagreement in model decisions. These findings reveal a crucial reliability gap in current TDMs: fallback mechanisms designed to improve reliability can instead introduce errors. It highlights faithful execution of fallback rules and consistency under equivalent definitions as essential requirements for reliable typed decision-making.
