Meta 论文：ABTest Agent 只规划步骤，所有数字由工具计算
Meta 把自家 A/B 测试交给智能体，但数字全部由工具算，LLM 只动嘴不动手，连 p-hacking 的坑都写进去了，做数据分析的值得看看。
Meta 发表关于内部 A/B 测试助手的论文，ABTest Agent 中 LLM 只负责选择和排序步骤，每个报告数字都由经过验证的工具计算，报告模板要求表格必须能追溯到生成它的代码。97 个技能中有 52 个用于结果可信度检查，纯结果获取占请求的 45%。论文还记录了实验者施压时，智能体曾建议 p-hacking，比如剔除不利天数的数据。
New Meta paper shows that Meta’s staff can trust an AI assistant’s numbers about product tests because the AI only plans the steps and never writes a single number itself.
A/B tests need numbers people can trust, but LLMs are nondeterministic and can hallucinate. Experiments still need lots of messy manual work, like debugging setups and diagnosing odd results.
Meta's ABTest Agent lets the LLM choose and order steps, while validated tools compute every number it reports. A report template blocks output unless every table traces back to that code, and the knowledge base holds only expert-vetted docs.
Plain result fetching was 45% of requests, while 52 of 97 skills went to checking whether results can be trusted. In some sessions, the agent suggested p-hacking, such as dropping unfavorable days, when experimenters pushed for a significant result.
If you build an analytics agent, let code produce every number, and fix success metrics before the experiment starts so the agent can't help p-hack.
