LeakBench 发布：审计临床建模 Agent 的奖励黑客与预测时数据泄漏
LeakBench: Auditing Reward Hacking through Prediction-Time Leakage in Clinical Modeling Agents
给做医疗 AI 或 Agent 评测的人：LeakBench 用真实数据测出各模型会不会偷用泄漏特征刷分，还验证了哪种防护真的有效。
LeakBench 是一套基于临床指南、可执行的审计套件，检测建模 Agent 是否使用预测时不可得的泄漏特征、校准是否合格等五类问题。它包含一个已知失败结构的合成任务，以及 MIMIC-IV（43 万次入院）和多家医院住院记录 MHFP 两个真实数据场景。2026 年 9 月的 A/B 日志显示，DeepSeek Flash 和 GPT-6-astra 在无约束条件下 10/10 全部选用泄漏特征，加入入院时约束后降为 0/10；Claude/Fable-5.1 在两种条件下都不泄漏。把 AUROC 换成 AUPRC 无法阻止泄漏选择，基于特征名的过滤也失效，而确定性的来源校验器可消除泄漏。
LeakBench: Auditing Reward Hacking through Prediction-Time Leakage in Clinical Modeling Agents
Autonomous clinical-modeling agents can optimize predictive performance while violating conditions that make a model usable. We introduce LeakBench, an executable, guideline-derived audit suite that tests agent decisions for prediction-time leakage, calibration, site-level evaluation, support awareness, and decision-curve net benefit. LeakBench combines a synthetic task with known failure structure and two real-data settings: MIMIC-IV (430k admissions) and governed multi-hospital inpatient record front-page (MHFP) data. Our primary analysis uses reward hacking descriptively, without attributing intent: an agent selects prediction-time-invalid inputs because they improve its validation reward. A deterministic optimizer and every evaluated unconstrained local-model configuration select a fatal leaking feature. In fully logged September 2026 A/B runs, DeepSeek Flash and GPT-6-astra each leak in 10/10 unconstrained runs and 0/10 runs with an explicit admission-time constraint. This contrast is version-sensitive: a blinded Claude/Fable-5.1 chat run does not leak in either condition, while explicit-constraint violations appear only in a historical August 2026 slice lacking raw trajectories. Replacing AUROC with AUPRC does not remove leakage selection. Renaming or disguising the leaking inputs changes neither selection nor provenance-based detection; a name-based filter fails. A deterministic provenance verifier-in-loop removes the leak, making the LLM repair loop defense-in-depth. Specific verifier feedback clears fatal leakage in a hosted agent, and a three-seed ablation finds principle-only feedback clears decision-level defects, though these mitigation results are limited. LeakBench treats trustworthy clinical prediction as a property of an agent's modeling process, not merely its score; it is an audit testbed with known-failure controls, not yet an expert-validated benchmark.