研究：15 个模型做企业级 Text-to-SQL 交付，仅 25% 通过全部验收
Can LLM Agents Be Forward-Deployed Engineers? Requirement Discovery and Delivery for Enterprise Text-to-SQL
这篇论文模拟了真实企业需求交付过程，发现最强的模型也只能发现八成需求，能全流程跑通的还不到三成，做企业落地的值得看看这四个坑。
arXiv 论文 2611.03551 测试 LLM 智能体能否担任前置部署工程师（FDE），在企业 Text-to-SQL 场景中自主发现需求并完成模型后训练交付。15 个模型中，表现最好的发现超过 80% 的隐藏需求，开源模型只发现 40-50%；交付阶段 8 种编码智能体配置中仅 25% 的运行通过全部验收标准。论文归纳出四个瓶颈，包括 40% 的需求缺失或被曲解、执行不稳定、算力预算利用不足，以及基准分数高不等于能上生产。作者建议对发现的需求做人工审核、监控执行中的计划，并写明包含硬件、预算和生产检查的实验环境规范。
Can LLM Agents Be Forward-Deployed Engineers? Requirement Discovery and Delivery for Enterprise Text-to-SQL
LLM coding agents can increasingly automate model post-training, but whether they can act as forward-deployed engineer (FDE) agents, discovering customer requirements and turning them into deployment decisions, remains unclear. We study this on a real enterprise text-to-SQL engagement: a simulated customer, grounded in the real use case's requirements, starts with an ambiguous request. An agent must discover acceptance criteria, data permissions, and hardware constraints in the discovery phase, then plan and execute a post-training recipe in the delivery phase. Across 15 models, the strongest discover over 80% of withheld requirements and open-weight models discover 40-50%; in the delivery phase, only 25% of runs across eight coding-agent configurations pass all acceptance criteria. We identify four bottlenecks in this engagement: (1) incomplete discovery, with 40% of withheld requirements missing or misrepresented; (2) unreliable execution; (3) underused compute budgets and a lack of systematic comparisons across base models and post-training methods; and (4) benchmark accuracy does not guarantee production readiness. We suggest human review of discovered requirements, monitoring of the agent's plan during execution, and an explicit experiment-environment specification covering hardware, budget scope, and production checks.
