SPY-Bench 基准：LVLM 推理时个性化隐私保护能力评估
Specify Privacy Yourself: Assessing Inference-Time Personalized Privacy Preservation Ability of Large Vision-Language Models
想让模型别在照片里乱说别人隐私？这篇论文证明现在几乎没人能做到——最强的 o4-mini 也只有约 24% 合规率。作者开源的 SPY-Bench 和对齐方法把准确率拉到 96.88%，做 VLM 安全或隐私产品的开发者直接能用。
arXiv 论文提出 ITP³（推理时个性化隐私保护）新任务：让用户通过语言指令动态指定隐私边界，而不是依赖模型内置的通用隐私规则。配套发布的 SPY-Bench 包含 32,700 个样本，覆盖 67 个类别和 24 个真实场景，并给出基于用户规格和上下文感知的新评估指标。
团队测试了 21 个 SOTA LVLM，发现即使表现最好的 o4-mini 合规准确率也只有约 24%，且模型的上下文隐私理解能力相当有限。这说明当前 LVLM 对通用隐私（如信用卡信息）对齐尚可，但对个性化、依赖语境的隐私需求几乎无法遵守。
作者随后实现了几种 ITP³ 对齐方法，其中 Noise Contrastive Alignment 变体把合规准确率提升到 96.88%，同时保持正常的通用性能。代码和数据已在 GitHub 开源。
Specify Privacy Yourself: Assessing Inference-Time Personalized Privacy Preservation Ability of Large Vision-Language Models
Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities but raise significant privacy concerns due to their abilities to infer sensitive personal information from images with high precision. While current LVLMs are relatively well aligned to protect universal privacy, e.g., credit card data, we argue that privacy is inherently personalized and context-dependent. This work pivots towards a novel task: can LVLMs achieve Inference-Time Personalized Privacy Protection (ITP$^3$), allowing users to dynamically specify privacy boundaries through language specifications? To this end, we present SPY-Bench, the first systematic assessment of ITP$^3$ ability, which comprises (1) 32,700 unique samples with image-question pairs and personalized privacy instructions across 67 categories and 24 real-world scenarios, and (2) novel metrics grounded in user specifications and context awareness. Benchmarking the ITP$^3$ ability of 21 SOTA LVLMs, we reveal that: (i) most models, even the top-performing o4-mini, perform poorly, with only $\sim$24% compliance accuracy; (ii) they show quite limited contextual privacy understanding capability. Therefore, we implemented initial ITP$^3$ alignment methods, including a novel Noise Contrastive Alignment variant which achieves 96.88% accuracy while maintaining reasonable general performance. These results mark an initial step towards the ethical deployment of more controllable LVLMs. Code and data are available at https://github.com/achernarwang/specify-privacy-yourself
本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：SPY-Bench、o4-mini、LVLM、AI安全，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.LG」（@Xingqi Wang, Xiaoyuan Yi, Xing Xie, Jia Jia），经 AITOP 多维评分管线结构化提炼。