论文

论文:QASecClaw 探究多智能体扩展在软件保障中的收益与局限

Beyond QASecClaw: Agent Scaling, Verifier Feedback, and Evidence Reliability in Software Assurance

精选理由

用 Qwen2.5-Coder 跑了五组实验,发现堆 50 个上下文 token 涨 49 倍却没效果,确定性工具证据反而更靠谱,做安全分析相关的可以看看。

该论文通过五组实验检验在已有确定性工具证据时,增加语言模型上下文数量能否提升软件保障效果,评估系统基于 Qwen2.5-Coder-7B-Instruct,共发出 13,638 次请求。在 8 个开发告警上,上下文数从 1 增加到 50,token 用量增长 49 倍,unsafe 清除率没有提升。在 32 个 SV-COMP 任务上,引入验证器反馈使平均正确判定从 12.7 升到 22.0,但错误判定也从 1 个增至 7 个。在两组各 120 个证据包测试中,模型拒绝 70-100% 的适用反例,而基于构造的范围检查全部分类正确,论文据此支持在测试场景下采用确定性证据准入。

原文 · arXiv cs.AI

Beyond QASecClaw: Agent Scaling, Verifier Feedback, and Evidence Reliability in Software Assurance

Can additional specialist language-model contexts improve software assurance when deterministic tool evidence is already available? We study this question through five experiments spanning authored Java warning controls, OWASP Benchmark assessment, Defects4J test generation, SV-COMP assertion verdicts, and deliberately misleading evidence packages. The evaluated systems use Qwen2.5-Coder-7B-Instruct, with a 1.5B sensitivity analysis, and complete 13,638 model requests. On eight related development warnings, increasing the context count from 1 to 50 increases token use 49-fold without improving unsafe clearance. On 160 OWASP cases, model configurations trade clearance against unresolved outcomes; the ten-context arm assesses only 96 cases. On 32 SV-COMP tasks, raw verifier feedback raises mean correct model verdicts from 12.7 to 22.0, but also raises incorrect verdicts from one to seven; a JBMC-plus-JVM validation baseline returns 26 correct verdicts and six unresolved outcomes. On two sets of 120 evidence packages, models reject 70-100% of applicable counterexamples, while construction-specific scope checks classify every package correctly. No target defect is discovered in three Defects4J bugs, one unsupported by the test adapter. These descriptive results support deterministic evidence admission for the tested settings, while exposing limits of verdict aggregation, feedback interpretation, and the available evaluation coverage.