研究：Verifier 选择是文生图基准的设计变量，会扭曲 Qwen-Image 与 FLUX.2 排名
Verifier Choice is a Benchmark Design Variable: Auditing Structural Counting Evaluation in Text-to-Image Models
做文生图评测或选型的人该看看这篇。它用 6240 张图实测 Qwen-Image、FLUX.2-dev、Z-Image，证明换个 verifier 协议连模型排名都能反转，连 GPT-5.5 数四叶草也只有约 60% 保真度。自己跑基准的话，文中那套 7 项 verifier 画像框架可以直接套用。
arXiv 论文 2611.15433 提出，自动评测分数并非只反映生成模型本身，还取决于 verifier、verifier-use 协议和打分规则。作者在结构计数这一可控任务上构建审计套件，覆盖手机、椅子、红绿灯等对象计数和四叶草叶片、抽屉面板、插座孔、涡轮叶片等子结构计数，共 7 类对象。
研究对 Qwen-Image-2512、FLUX.2-dev 和 Z-Image 三个模型生成 6,240 张图片，比较多种开源 MLLM verifier 的 count-only、reject-aware、zero-shot 与 few-shot 协议及 TIFA 式查询和检测器基线。在 2,280 张均衡评测集上，总体分数接近的配置在图片级仍大量不一致；每种 verifier-协议配置都会把两模型的分数差相对人工结果缩小 7.5-25.0 个百分点，且全部 4 种协议都会在固定提示集上逆转 Z-Image 与 FLUX.2-dev 的人工排序。
闭源前沿模型同样不保险：GPT-5.5 在四叶草上的有效计数保真度仅 59.92%，Claude Opus 4.7 同切片一致性 κ=0.432。作者建议报告自动基准分数时附带分解式 verifier 可靠性画像，包括图片级一致性、模型对比扭曲度、有效计数保真度等 7 项指标。
Verifier Choice is a Benchmark Design Variable: Auditing Structural Counting Evaluation in Text-to-Image Models
Automatic evaluation is now central to text-to-image benchmarking, but the verifier that converts generated images into scores is often treated as an implementation detail. We argue that this is unsafe: an automatic benchmark score is a property of the image generator, prompt distribution, verifier, verifier-use protocol, and scoring rule, not of the generator alone. We study this issue in structural counting, a controlled setting where human-visible counts can be audited. We construct a verifier-centric audit suite over seven counting objects: cell phones, chairs, and traffic lights for object counting, and clover leaflets, dresser drawer fronts, power-strip sockets, and turbine blades for substructure counting. Across 6,240 generated images from Qwen-Image-2512 and FLUX.2-dev, we compare open MLLM verifiers under count-only, reject-aware, zero-shot and few-shot protocols, plus TIFA-like target queries, a detector baseline, and frontier closed-model checks on two hard objects. On a balanced 2,280-image evaluation split, aggregate score matching can coexist with image-level disagreement, and verifier-use protocols and generator distributions change the measured reliability. Every verifier-protocol configuration also shrinks the human score difference between the two generators, by 7.5-25.0 pp, and with a third generator, Z-Image, all four tested protocols reverse the human ordering of Z-Image and FLUX.2-dev on our fixed prompt suite. GPT-5.5 and Claude Opus 4.7 checks do not remove the failure modes: GPT-5.5 still has only 59.92% clover valid-count fidelity, while Claude Opus 4.7 has lower same-slice agreement ($κ=0.432$). We propose reporting automatic benchmark scores with a factorized verifier reliability profile: aggregate score gap, image-level agreement, generator-comparison distortion, valid-image count fidelity, rejection behavior, output compliance, and directional bias.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：text-to-image、Qwen-Image、FLUX.2、benchmark，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.LG」（@Shurun Li, Michael Hua Wang, Haibo Zeng, Long Wang），经 AITOP 多维评分管线结构化提炼。