SurgKnowBench 与 SurgAgentGym：评测通用智能体的手术场景理解能力
How to "train" a surgeon? Benchmarking generalist agents in surgical scene understanding
如果你在做医疗多模态应用，这篇论文把现有 MLLM 在手术场景的短板摸得很清楚：感知任务靠工具调用能补上，但流程评估和细粒度定位还差很远。两套评测框架可以直接拿来当自家模型的压力测试。
一篇新论文提出两阶段评估框架，用于检验通用多模态大语言模型（MLLM）在手术场景理解上的真实水平。第一阶段推出诊断基准 SurgKnowBench，统一整合公开手术数据集，覆盖多种术式与任务，对专有和开源 MLLM 进行系统评测。结果显示现有模型保留部分视觉先验，但在关系与流程理解、工作流评估上表现薄弱。
第二阶段在此基础上构建了智能体环境 SurgAgentGym：模型可以在推理时对 SurgWiki 进行有界知识检索，并调用通用视觉定位工具来完成手术任务，全程无需额外的手术领域训练。核心问题是验证工具化的证据使用能否让通用智能体即时提升手术理解能力。
结论是工具调用主要在感知密集型任务上缩小差距，而细粒度定位和专家级评估仍是持续瓶颈。这为智能体化手术理解研究提供了可复用的路径，对医疗 AI 团队和手术机器人方向的研究者有直接参考价值。
How to "train" a surgeon? Benchmarking generalist agents in surgical scene understanding
Evaluating general-purpose multimodal large language models (MLLMs) for surgical scene understanding remains limited due to scarce surgical data and fragmented evaluation protocols. We address these challenges through a two-stage evaluation framework for surgical scene understanding. In Stage I, we introduce SurgKnowBench, a diagnostic benchmark that harmonizes public surgical datasets across procedures and tasks and evaluates proprietary and open-source MLLMs to map the boundary of pretrained surgical competence. SurgKnowBench reveals that current models retain partial visual priors but remain weak on relational and procedural understanding and workflow assessment. Building on this failure frontier and motivated by the scaffolded nature of surgical training, we introduce SurgAgentGym, an agentic environment where general-purpose MLLM agents can use bounded knowledge search over SurgWiki and general-purpose visual grounding for surgical tasks. SurgAgentGym evaluates whether generalist agents can improve surgical understanding at inference time through tool-mediated evidence use, without additional surgical training. Our results show that agentic inference narrows the gap mainly for perception-heavy tasks, while fine-grained localization and expert-style assessment remain persistent bottlenecks. This opens a path toward future studies of agentic surgical understanding.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：SurgKnowBench、SurgAgentGym、多模态、智能体，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.AI」（@Gengyuan Zhang, Xiao Han, Shijie Zhou, Abdullah Koukash, Jingpei Wu, Jindong Gu, Yuqicheng Zhu, Nassir Navab, Volker Tresp, Ghazal Ghazaei），经 AITOP 多维评分管线结构化提炼。