Harvey 用 context graphs 提升 AI agent 法律任务表现
LLM-based agents love structured data. I've seen this work really well, but mostly in complex code...
Harvey 证明给 agent 建结构化的 context graphs，法律任务准确率能涨十几二十个点，成本还省 48%，做 agent 的可以研究下。
Harvey 发布报告，介绍用 context graphs 帮助 agent 处理长周期法律任务。agent 在不接触评测任务和评分标准的情况下，对文档构建结构化图谱，覆盖 Legal Agent Bench 11 类任务，包括 M&A 尽调和合同谈判。结果上，Firm Knowledge 任务从 65% 升到 80%，Matter counting 任务从 53% 升到 83%，agent 成本下降 48%。dair-ai 也计划把该方法应用到科研任务上。
LLM-based agents love structured data. I've seen this work really well, but mostly in complex code...
LLM-based agents love structured data. I've seen this work really well, but mostly in complex codebases. But I think context graphs have broader applications, as shown in this Harvey report on legal knowledge tasks. We are working on this at @dair_ai for research-related tasks. More to share soon. Harvey @harvey We built context graphs to help agents work through long-horizon legal tasks. Starting with Legal Agent Bench tasks in 11 categories like M&A diligence and contract negotiation, we had agents build context graphs of the underlying documents (without access to the eval tasks or grading rubrics). Context graphs dramatically improved agent performance on LAB tasks: - Firm Knowledge tasks: 65% → 80% - Precedent lists tasks: 67% → 80% - Matter counting tasks: 53% → 83% They also boost agent efficiency: we found a 48% cost reduction for agents given access to a context graph to assist their work. Full video below - more to come soon. 🔗 View Quoted Tweet 💬 6 🔄 0 ❤️ 6 👀 985 📊 5 ⚡
