InclusionMed:面向包容性医疗的 AI 评测基准体系发布
InclusionMed: Benchmarking AI for Inclusive Healthcare
一篇医疗 AI 评测基准论文,从 388 个候选里筛出 5 个靠谱基准,测下来顶级模型总分都没过 50 分。做医疗 AI 选型的团队可以直接拿来当选型参考,做评测的研究者也能从它的审计和复现流程里学到东西。
研究团队在 arXiv 发布 InclusionMed,一个面向包容性医疗的开放 AI 评测计划。其 Atlas 框架按 3 种医疗角色和 3 种 AI 能力划分 9 个格子,首批从 388 个候选基准中经结构化审计与独立复现选出 4 个公开基准,加上私有诊断推理基准 InclusionMed-DxFiles,共覆盖 5 个基准、9 格中的 3 格。
团队在 Atlas 上评测了多个前沿大语言模型,结果汇总为 InclusionMed Index,总分范围在 17.5 到 49.1(满分 100)之间。Index 还按医疗角色和 AI 能力分别打分,发现模型排名会随角色或能力维度变化。任务级标签分析显示,Index 排名前 10 的模型在患者沟通上的得分均高于临床决策。总分相近的模型,其估算推理成本差异也很大。
论文指出当前基准覆盖仍不均衡,呼吁以更多元的医疗需求来定义和衡量医疗 AI 进展,并通过社区开放协作扩展评测覆盖。
InclusionMed: Benchmarking AI for Inclusive Healthcare
Advancing medical AI for inclusive healthcare requires evaluation across diverse healthcare roles, tasks, and care settings, but benchmark coverage remains uneven. We introduce InclusionMed, an open medical AI evaluation initiative for inclusive healthcare. The InclusionMed Atlas organizes benchmarks across three healthcare roles and three AI capabilities. For the first release, we selected four public benchmarks from 388 candidates through structured audits and independent reproduction. With InclusionMed-DxFiles, our private diagnostic reasoning benchmark, the Atlas contains five benchmarks covering three of its nine role--capability cells. We evaluated frontier large language models on the Atlas and aggregated their results into the InclusionMed Index, with overall scores ranging from 17.5 to 49.1 out of 100. The Index also reports scores by healthcare role and AI capability, showing that a model's rank varied with the role or capability considered. A within-benchmark analysis using task-level tags showed that the ten highest-ranked models in the Index all scored higher on patient communication than on clinical decision-making. Models with similar Index scores also differed substantially in estimated inference cost. These findings highlight the need for evaluation covering more diverse healthcare needs and for context-specific model selection. Through open collaboration and community contributions, InclusionMed seeks to ensure that diverse healthcare needs worldwide shape how medical AI progress is defined and measured.
本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:InclusionMed、医疗AI、基准评测、临床决策,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@ The InclusionMed Collaboration),经 AITOP 多维评分管线结构化提炼。