BMMS 评测套件：按具体政策条款衡量语言模型行为合规性
BMMS: Policy-Conditioned Measurement of Behavioral Compliance in Language Models
这篇论文做了个叫 BMMS 的合规评测：把 OpenAI、Anthropic 自己写的规范当考卷，结果 GPT-4o 70.7% 的硬约束没过，Claude 几乎全过，挺有意思。
arXiv 论文提出 Behavioral Misalignment Metric Suite（BMMS），以任意政策文本为输入，输出该系统的合规画像，核心指标包括 Compliance Attainment Score（CAS）与 Compliance Robustness Margin（CRM）。指标需通过以归因为判据的准入测试，套件据此剔除了自身的一个指标。在来自 11 份文档的 1,003 条义务上，准入判据对厂商规范的盲评精确率达 94.4%，对分析类指南为 50.0%。实测显示，GPT-4o 在 OpenAI Model Spec 的硬性约束上失败 70.7%，而两款 Claude 模型分别失败 0.0% 和 9.1%。
BMMS: Policy-Conditioned Measurement of Behavioral Compliance in Language Models
Behavioral evaluation of language models fixes its dimensions in advance: hallucination, sycophancy, toxicity, over-refusal. An operator in a regulated industry needs a different question answered: does a system do what a particular rulebook requires? We introduce the Behavioral Misalignment Metric Suite (BMMS), which measures policy-conditioned misalignment by taking a policy corpus as an argument and returning a compliance profile over it, for any published policy and any system that answers in natural language. The Compliance Attainment Score (CAS) measures severity-weighted attainment over the obligations that bind the system, and the Compliance Robustness Margin (CRM) the share of that attainment surviving graded adversarial pressure; Behavioral Invariance (BIV) and the Behavioral Disposition Index (BDI) are specified and released with the suite. Dimensions enter only by passing a published admission test whose binding criterion is attribution: a quantity that moves when the judge changes, holding the target fixed, is measuring the instrument. Applying it excluded one of the quantities the suite itself reported. Across 1,003 obligations from eleven documents in five registers, the admission criterion holds 94.4% precision against blinded human judgment on vendor specifications and 50.0% on analytical guidance, extraction fidelity is 94.6%, and re-scoring under judges from two other providers leaves target ordering invariant while the absolute score of a single target spans up to 0.22. We apply it to the two most detailed published behavioral specifications. Own-specification advantage is not resolved from zero once document difficulty is subtracted, but absolute self-compliance is another matter: GPT-4o fails 70.7% of the hard constraints binding it in the Model Spec published by OpenAI, where the two Claude models fail 0.0% and 9.1% of theirs.