Trains but Doesn't Learn：面向 LLM 智能体交付能力的 Post-Training 基准
Trains but Doesn't Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers
这篇论文造了个很较真的基准：不看智能体能不能把指标刷上去，只看它交付的模型到底行不行，连"训练了但没学到东西"这种静默失败都能抓住，四个前沿模型全被拉出来和人类工程师对比。
论文提出 Post-Training as a Service（PTaaS）场景下的交付基准，考察 LLM 智能体能否作为 forward-deployed engineer 在预算与人工审批约束下完成微调、评估和部署全流程。基准包含十个阶段，由 oracle 基于平台记录的事实逐段打分，覆盖 8B 到 70B 的开源基座模型。核心失败模式是 TBDL：训练损失下降、各信号正常，但交付模型相比基座没有提升；operator 运行的验收门能在付款前拦截所有此类运行。评测跑了 Claude Opus 5、GPT-5.6-luna、Gemini 3.7 Flash、DeepSeek V4-Pro 四个模型，在 L40S、A100、H200 GPU 上端到端执行，并设置了人类 FDE 对照组。
Trains but Doesn't Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers
Post-training is becoming a service (PTaaS): a customer hands an operator data and a goal, and a forward-deployed engineer (FDE) returns a fine-tuned, evaluated, and deployed model under a budget, a human-approval gate, and reproducibility requirements. Seating an LLM agent in the FDE seat raises a question existing benchmarks cannot answer: not whether an agent can raise a metric, but whether it can be trusted to deliver. We answer it on a governed delivery plane, where an agent drives ten stages and an oracle scores each stage from platform-recorded facts. The central silent failure is the run that trains but does not learn (TBDL): loss falls, every signal stays green, and the delivered model is no better than the base. An operator-run acceptance gate catches every such run before payment, and a detector calibrated on known-corrupted runs flags severe corruption mid-run. We ran four frontier agents (Claude Opus 5, GPT-5.6-luna, Gemini 3.7 Flash, DeepSeek V4-Pro) end to end on metered L40S, A100, and H200 GPUs across 8B to 70B open bases, certifying every scenario before scoring. We also ran a human FDE arm under the same oracle and compare every agent against it.
