论文

小参数编程 Agent 用不好代码图工具,arXiv 论文实测 79k 次试验

How Do Small Coding Agents Use Code Graphs and Embeddings?

精选理由

79k 次实验说清一件事:小模型配代码图工具,增益集中在一跳查找,多跳基本用不起来。

论文在 SWE-bench Verified 上测试小参数开源模型使用嵌入检索与代码图工具的效果,覆盖 Qwen3 0.6B 到 32B 六个尺寸和三个 Gemma 4 尺寸,共 79k 次试验。结构化工具让 Qwen3-8B 和 Gemma 4 E4B 的 grep-only 准确率翻倍,增益在 1.7B-4B 达到 17-21 个百分点,到 32B 降至 4-5 个百分点。每增加一跳代码图深度,正确定位的几率乘以 0.50-0.67,任何工具组合都无法抹平这一梯度。线性探针显示小模型对仓库的表征主要是一跳且依赖词面的信息,直接调用边解码 AUC 为 0.74,低于词面基线的 0.99。论文已开源 Agent、探针、日志与轨迹。

原文 · arXiv cs.LG

How Do Small Coding Agents Use Code Graphs and Embeddings?

Coding agents are given embedding search and code-graph tools on the premise that a repository's structure is something the model lacks and the tool supplies. We test that premise on SWE-bench Verified with small open-weight models, stratifying tasks by a deterministic hop depth (the shortest code-graph path from code the issue mentions to code the gold patch changes) that a companion paper defines and validates. First, we ablate a localization agent's tool set (grep only; +embedding search; +graph tools; all) on a depth-balanced 160-task split and on all 500 tasks for ten models (six Qwen3 sizes from 0.6B to 32B with five decodings each, an FP8 control, three Gemma 4 sizes; 79k episodes). Structured tools double grep-only accuracy for Qwen3-8B and Gemma 4 E4B; their gain peaks at 17-21 points at 1.7-4B and falls to 4-5 points at 32B (odds ratio 0.77-0.82 per doubling of parameters). From 8B up, about half of that gain is the form of the submission rather than finding the target; the fall from 14B to 32B is a fall in tool uptake at equal grep-only accuracy; and the same 4B size gains or does not gain from a tool depending on the checkpoint. Each extra hop multiplies the odds of correct localization by 0.50-0.67 at every size, and no tool set flattens that gradient: traces show the graph tools' gain is name lookup at hop 0 and one expansion at hop 1, and agents chain expansions in only 12-14% of episodes. A fixed repair model resolves 6-13 of 160 tasks from these localizations (oracle 26). Second, linear probes find direct call edges decodable under a within-anchor, degree-stratified protocol (AUC 0.74; 0.61-0.66 inside agent conversations), but below a lexical baseline (0.99), with two-hop edges at 0.57. Small agents carry a one-hop, largely lexical picture of their repository, in weights and in use. We release the agent, probes, logs and traces.

  • TestingCatalog10-25 17:06原文