IndustrialKnowEval:覆盖43,282样本的工业知识评测基准发布
IndustrialKnowEval: A Benchmark for Evaluating Multi-Level Industrial Knowledge in LLMs
工业领域的知识问答一直是模型短板,这个基准把认知分成四层来测,还专门设计了衡量基础知识和工程应用落差的指标。做工业AI选型或垂直微调的团队可以拿去跑一遍自家的模型。
IndustrialKnowEval(IndKnow)是一个面向大语言模型工业知识与工程问答能力的评测基准,包含43,282个样本,覆盖21种任务类型和七个工业领域,涉及航空航天、电力、冶金等方向。数据集按认知需求分为记忆、理解、推理、工程应用四个层级,企业领域专家和资深研究人员参与了从数据构建到质量控制的全部环节。研究团队在18个代表性大模型上开展了系列实验,并由领域专家深度参与评测过程。
基于评测结果,论文提出工业认知能力差距(ICCG)指标,用于刻画模型在基础工业知识与高阶推理、工程应用之间的性能落差,并结合绝对性能对模型能力进行更完整的分析。此前针对工业场景的评测基准较少,多数基准偏向通用知识或编码能力,IndKnow 补充了按认知层级细分工业知识的评测路径。代码已在 GitHub 开源(github.com/J4cksonLee/IndKnow),可支持工业智能化方向的模型选型与基准测试。
IndustrialKnowEval: A Benchmark for Evaluating Multi-Level Industrial Knowledge in LLMs
Large language models (LLMs) are increasingly explored for industrial tasks, yet few benchmarks evaluate their mastery of industrial knowledge across different types of cognitive demand. We introduce IndustrialKnowEval (IndKnow), a benchmark for industrial knowledge and engineering question answering. It contains 43,282 samples covering 21 task types across seven industrial sectors, including aerospace, electric power, and metallurgy, organized into four levels of cognitive demand: memory, comprehension, reasoning, and engineering application. Enterprise-based domain specialists and experienced researchers participate in all stages of dataset construction and quality control. We conduct a series of experiments on 18 representative LLMs, with domain experts deeply involved in the evaluation. Building upon IndKnow, we propose the Industrial Cognitive Capability Gap (ICCG) metric to characterize the performance differential between foundational industrial knowledge and higher-order reasoning and application, and we analyze it alongside absolute performance to provide a more complete view of model capabilities. We anticipate that IndustrialKnowEval will support the benchmarking of LLMs in industrial scenarios and promote the development of robust industrial intelligence. Code is available at: https://github.com/J4cksonLee/IndKnow
本条情报归属 AITOP「论文」核心栏目,重点聚焦中国 AI 产业与国产大模型生态。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:IndustrialKnowEval、LLM评测基准、工程问答、工业智能,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Haiteng Wang, Haoze Li, Jingwei Guo, Jingbo Zhou, Si Liu, Lei Ren),经 AITOP 多维评分管线结构化提炼。