做LLM智能体系统设计的工程师和研究者会看到具体数据:什么设计真正有效、什么只是烧token。这篇论文给出了可操作的优先级——先做好状态抽象和任务分解,再考虑推理增强,值得点开对照自己的设计。
该研究在CybORG CAGE-2(一个对抗性部分可观测环境)中系统评估了复合LLM智能体的设计维度:上下文表示、推理方式和任务分解。实验涵盖5个模型家族、6个模型和12种配置,共3475个回合,并进行了token级成本核算。主要发现包括:程序化状态抽象比原始观测提升最多76%的回报;将推理工具分布在层级中会导致性能下降(最多3.4倍),同时增加1.8-2.7倍token消耗,称为“推理级联”;无推理的层级分解在大多数模型上取得最佳绝对性能。研究建议在结构化对抗POMDP中优先投资程序化基础设施和清晰任务分解,而非加深每个智能体的推理。
Context, Reasoning, and Hierarchy: A Cost-Performance Study of Compound LLM Agent Design in an Adversarial POMDP
Deploying compound LLM agents in adversarial, partially observable sequential environments requires navigating several design dimensions: (1) what the agent sees, (2) how it reasons, and (3) how tasks are decomposed across components. Yet practitioners lack guidance on which design choices improve performance versus merely increase inference costs. We present a controlled study of compound LLM agent design in CybORG CAGE-2, a cyber defense environment modeled as a Partially Observable Markov Decision Process (POMDP). Reward is non-positive, so all configurations operate in a failure-mitigation mode. Our evaluation spans five model families, six models, and twelve configurations (3,475 episodes) with token-level cost accounting. We vary context representation (raw observations vs. a deterministic state-tracking layer with compressed history), deliberation (self-questioning, self-critique, and self-improvement tools, with optional chain-of-thought prompting), and hierarchical decomposition (monolithic ReAct vs. delegation to specialized sub-agents). We find that: (1) Programmatic state abstraction delivers the largest returns per token spent (RPTS), improving mean return by up to 76% over raw observations. (2) Distributing deliberation tools across a hierarchy degrades performance relative to hierarchy alone for all five model families, reaching up to 3.4$\times$ worse mean return while using 1.8-2.7$\times$ more tokens. We call this destructive pattern a deliberation cascade. (3) Hierarchical decomposition without deliberation achieves the best absolute performance for most models, and context engineering is generally more cost-effective than deliberation. These findings suggest a design principle for structured adversarial POMDPs: invest in programmatic infrastructure and clean task decomposition rather than deeper per-agent reasoning, as these strategies can interfere when combined.