论文

VLM 小目标识别研究:何时缩放、何时切图最有效

Know When to Zoom: How Small Is Too Small for Vision-Language Models?

精选理由

做 Agent 或处理卫星图、无人机图的人真的该看这篇。它用具体数字告诉你各家 VLM 丢细节的程度,还给出省 token 的切图触发策略,比无脑切九宫格省 30% 以上成本。

一项 arXiv 研究(编号 2611.22709)测量了来自四家厂商的 7 个视觉语言模型在大图上的小细节识别能力。测试发现,一张 8000×6000 的无人机照片中宽 20 像素的汽车,到达 Claude 时只剩约 6 像素宽,而 OpenAI 默认设置会直接拒绝这张照片。在 7680×4320 照片上,模型能读到一半正确率的文本尺寸从 6 像素以下升至预测的 10 到 13 像素。

正文

把照片切成最多 9 块图块(tiles)能大幅挽回细节:Claude 和 Gemini 的准确率提升 11.1 到 14.5 个百分点,GPT-6.1 Sol 因避免了拒绝,从 50.7% 升到 83.7%。两步缩放——先让模型指出需要查看的区域——在 Gemini 上持平或超过切图,在 Claude 上差距 2.5 个百分点以内,且只消耗切图约 40% 的图像 token。失败模式也有差异:细节丢失时,Gemini 在 43% 到 48% 的情况下给出自信的错误答案,Claude 只有 5% 到 9%。

基于细节尺寸的预检查会漏掉 Claude 和 Gemini 上切图的大部分收益,而 Claude 自身的不确定回复是可靠触发条件:仅在模型回答不确定后再切图,效果与总是切图相差 1.4 个百分点以内,token 消耗降到 62% 到 70%。团队据此总结了 5 条 Agent 缩放规则,并公开了视力表测试、全部模型答案和代码。

原文 · arXiv cs.AI

Know When to Zoom: How Small Is Too Small for Vision-Language Models?

Online vision-language models often miss small details in large images, because the service shrinks the image, or rejects it, before the model ever sees it. A car 20 pixels wide in an 8000 by 6000 drone photo reaches Claude about 6 pixels wide, and OpenAI's default rejects the photo outright. We measure how much detail survives for seven models from four vendors, using an eye exam of printed codes and 601 questions about small objects in real photos. The smallest text a model reads half the time grows from 6 pixels or less on a small image to a predicted 10 to 13 pixels on a 7680 by 4320 photo at default settings. Zooming wins much of the detail back. Splitting each photo into up to nine tiles raises accuracy by 11.1 to 14.5 percentage points for Claude and Gemini, and by avoiding rejections it lifts GPT-6.1 Sol from 50.7% to 83.7%. A two-step zoom, in which the model first names the region it needs, matches or beats tiles on Gemini and comes within 2.5 points of them on Claude, with about 40% of their image tokens. When a detail is lost, Claude and Gemini both say they cannot tell about half the time, but they fail differently in the rest: Gemini gives a confident wrong answer in 43% to 48% of these cases, Claude in only 5% to 9%. A check based on detail size, run before the call, misses most of the gain from tiles on Claude and Gemini, but Claude's own reply is a reliable trigger: tiling only after an unsure reply comes within 1.4 points of always tiling with 62% to 70% of the image tokens. We turn these findings into five rules for when an agent should zoom, and release the eye exam, every model answer and code at https://siddvoh.com/know-when-to-zoom.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:VLM、Gemini、Claude、GPT-6.1 Sol,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Siddharth Vohra, Min Xu),经 AITOP 多维评分管线结构化提炼。