研究：LLM 在恶劣天气提醒场景中的错误安抚问题与可复现评测
False Reassurance in LLM Severe-Weather Guidance: A Replayable Evaluation Against Real NWS Alerts
用真实 NWS 警报测 Claude 天气管道，发现盲测下近四成回答会错误安抚用户，还放出了数据集和代码可以自己复现。
一项研究通过回放 117 个基于美国国家气象局（NWS）真实警报构建的冻结案例，评测基于 Claude 的天气预警管道是否会给用户错误的安抚。结果显示：生产管道在 66 次警报场景中错误安抚率为 0.0%，盲测中性提示词下为 37.7%（26/69），而要求模型自行披露盲区并不能解决问题（p=0.40）。研究还发现，仅有警报数据并不足以保证安全行为：中性提示词在 87.5% 的无警报对照组中触发了误报警报。团队公开了数据集、标注与代码，并指出 NWS 测试产品被误当真实警报摄取、token 预算不足等常见缺陷。
False Reassurance in LLM Severe-Weather Guidance: A Replayable Evaluation Against Real NWS Alerts
An LLM assistant that cannot see official severe-weather alerts may tell users that conditions are calm while a warning is active. We call this false reassurance and measure it by replaying 117 frozen cases built from real U.S. National Weather Service (NWS) alerts (69 alert cases, 32 no-alert controls, 16 non-U.S. locations) through five conditions of a Claude-based preparedness pipeline that differ in alert access and prompt design. One rater hand-labeled all 342 alert-case responses under a seven-category scheme written after an initial binary pass, which we disclose. False reassurance was 0.0% for the production pipeline (0/66, 95% CI 0.0-5.5) and for the same alerts under a neutral prompt (0/69), 37.7% for a blind neutral prompt (26/69), and 44.9% when the blind prompt also told the model to disclose its blind spot (31/69). Every case was implicit; no response explicitly denied an alert. Real alerts removed false reassurance (paired McNemar p < 0.0001); the disclosure instruction did not (p = 0.40). An automated regex measure suggested the opposite and missed 53 of 57 true cases, so we do not use it. Alert data alone was not enough for safe behavior: with a neutral prompt the model raised some alarm on 87.5% of no-alert controls, against 0% under the production prompt, and a preregistered single-sentence ablation showed that one production instruction explains part of that gap. We also report two failures common to similar systems, NWS test products ingested as real alerts and a token budget that fails on complex cases, and we release the dataset, labels and code.
