论文

给模型一根手指:用儿童计数策略审计 VLM 的视觉推理

Give the Model a Finger: Developmentally Aligned Evaluation of Counting in Vision-Language Models

精选理由

研究者让 VLM 像小孩学数数那样逐个点物品数数,9 个模型的精确计数范围差了近 100 倍,标记还能审计模型是不是真在看图。

论文把儿童学会数数时的动作——逐个指点、移开已数物品、划分区域——作为无视觉能力的哑图像工具,嵌入免训练的 agent 循环中测试 9 个视觉语言模型。实验覆盖 682 张图片、7 种测试条件,发现各模型的精确计数范围差异近两个数量级(R80 从 3 到 150 个点)。作为干预手段,指点标记只在 9 个模型中的 4 个小数量照片场景有帮助,很少胜过纯语言流程;但作为测量手段,标记扰动探针能揭示模型是否真的从标记读出答案、指点精度与收益的关联,以及密集场景主要因漏数而失败。

原文 · arXiv cs.LG

Give the Model a Finger: Developmentally Aligned Evaluation of Counting in Vision-Language Models

Vision-language models (VLMs) are exact on small sets and drift into estimation on larger ones, a profile reminiscent of children before they master counting. Children cross the barrier with epistemic actions: pointing at each item while reciting number words, moving counted items aside, partitioning the array. We give VLMs the same actions as dumb image tools with no visual intelligence of their own in a training-free agent loop. Our central claim is about measurement, not intervention: a model that counts by marking leaves a normally hidden serial procedure on the image, and the developmental principles of counting turn those marks into an interpretable audit of serial visual reasoning: every tag is classified against point-level ground truth as on-target, duplicate, phantom, or omitted, the answer is checked against the number of tags, and order irrelevance is tested by forcing opposite traversal orders. Across 9 models, 682 images, and 7 harness conditions we find that (i) the exact-counting range is a model-specific signature spanning nearly two orders of magnitude (R80 from 3 to 150 dots); (ii) as an intervention the finger mostly fails: tags help 4 of 9 models on small-count photographs, hurt imprecise pointers, and rarely beat a verbal routine or a blind point list; what the loop buys is an auditable trace, not accuracy; (iii) as measurement it succeeds: a tag-perturbation probe shows that precise pointers read their answers off the tags, pointing precision tracks which models benefit, dense scenes fail by omission, and models rarely revise after seeing their own tags; and (iv) the capacities dissociate: models follow an instructed traversal order yet often change their answer when it reverses, and children's regular-over-scattered advantage appears only under the routine and survives lattices that defeat a rows-times-columns shortcut.