FG-GraphBench 基准发布:用多模态场景图评测细粒度视觉理解
FG-GraphBench: Benchmarking and Enhancing Fine-Grained Visual Understanding with Multimodal Scene Graphs
搞多模态的研究者可以关注,这篇同时给了基准和方法:FG-GraphBench 拆成三个任务测细粒度理解,FiGRA 把场景图编码成图 token 喂进 MLLM,三个任务都跑赢了强基线。
一项新研究提出了 FG-GraphBench(Fine-Grained Multimodal Scene Graph Benchmark),一个针对多模态大语言模型细粒度视觉理解能力的基准,包含三个任务:细粒度视觉问答(FG-QA)、图像描述生成(FG-Caption)和图像描述(FG-Description)。基准采用两阶段构建流程,先发现并定位细粒度视觉实体,再建立实体间关系,生成多模态场景图(MSGs),以节点表示实体、边表示关系来组织视觉信息。研究指出两个核心问题:现有场景图基准对细粒度实体及其关系的覆盖不足,且现有方法未能在 MLLM 中有效利用图结构信息。
针对这些问题,作者提出 FiGRA(Fine-grained Graph Representation Augmentation)方法,采用两阶段训练策略,将多模态实体和关系特征编码为连续的图 token 并注入 MLLM。实验结果显示,FiGRA 在 FG-GraphBench 的三个任务上均显著优于强基线模型,进一步分析证实了保留细粒度实体和关系信息对视觉理解的重要性。
FG-GraphBench: Benchmarking and Enhancing Fine-Grained Visual Understanding with Multimodal Scene Graphs
Fine-grained visual understanding requires multimodal large language models (MLLMs) to recognize small visual entities and understand their relationships. Scene graphs offer a natural way to represent this information, organizing visual entities as nodes and their relationships as edges. However, we identify two challenges: existing scene graph benchmarks provide insufficient coverage of fine-grained entities and the relationships among them, and existing methods for fine-grained visual understanding have yet to effectively leverage such graph-structured information in MLLMs. To address these challenges, we introduce Fine-Grained Multimodal Scene Graph Benchmark (FG-GraphBench) and FiGRA. FG-GraphBench is a benchmark comprising three complementary tasks: fine-grained visual question answering (FG-QA), image captioning (FG-Caption), and image description generation (FG-Description). We develop a two-stage construction pipeline that discovers and grounds fine-grained entities and establishes their relations to construct the fine-grained multimodal scene graphs (MSGs) in FG-GraphBench. Building on these MSGs, we propose FiGRA, a Fine-grained Graph Representation Augmentation method for MLLMs. FiGRA adopts a two-stage training strategy. It encodes multimodal entity and relation features as continuous graph tokens, and integrates them into the MLLM. Experiments show that FiGRA significantly outperforms strong baselines across three tasks of FG-GraphBench. Further analyses confirm the effectiveness of FG-GraphBench and highlight the importance of preserving fine-grained entity and relation information for visual understanding.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:FG-GraphBench、FiGRA、多模态、场景图,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Sicheng Liu, Xunkai Li, Ziyu Han, Xingnan Zhang, Ru Zhang, Rong-Hua Li),经 AITOP 多维评分管线结构化提炼。