前Anthropic安全团队负责人离开，批评AI公司安全投入不足

他举例HuggingFace事件，认为对于可能带来灭绝级风险的科技，这种缺乏透明度是不可接受的。他主张公众应要求更多透明度，并建议公司应披露其递归自我改进的进展、报告安全事件和近失事件，并接受独立安全标准验证。

“right now, AI companies are underinvesting in safety” might prove to be the understatement of the century. Joe Benton @JoeJBenton I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why. AI companies are racing to build machines that are much smarter than any human, and we may not survive this. I want to work from the outside to ensure the public is informed about these risks, and help the world navigate this transition responsibly. Right now, AI companies are underinvesting in safety. A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing. We only found out about the HuggingFace incident because the agents broke out onto the public internet. I don’t think that’s acceptable for a technology that might cause extinction-level risks. The public should demand far more transparency. We can’t steer this technology safely without more people being able to see where it’s going. Some of this is basic: companies should disclose their progress towards recursive self-improvement, report safety incidents and near-misses, meet minimum safety standards, and get independent guarantees that they are meeting those standards. I’ll be joining @METR_Evals to do independent evaluations of these risks. I want to show the world that these guardrails are possible, and that by doing them we can move these companies’ incentives away from racing and towards responsible development. I wrote up more thoughts here on my decision and what I hope changes: substack.com/@jbenton1/p-21… 🔗 View Quoted Tweet📊 8 ⚡

本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「各国 AI 监管法案、合规治理与主权算力政策」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：Anthropic、AI安全、递归自我改进，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明：本文原始线索采编自「Gary Marcus」（@GaryMarcus），经 AITOP 多维评分管线结构化提炼。截至目前，全网已有 11 家独立信源对该事件进行交叉印证与跟进报道（含 Paul Graham、Decoder、IT之家 等），属于近期高关注度行业信号。