Latent-Lens：让小型语言模型通过隐空间直接接收视觉信息
Latent-Lens: Visual Perception in Small Language Models Through Latent Communication
一篇有意思的论文：VLM 和小模型不再转文字传话，直接在隐空间传视觉状态，只训 0.2% 的参数，CLEVR 上比文本通信高出 22 分。
arXiv 论文 Latent-Lens 提出一种轻量级编解码器，让冻结的 VLM 与冻结的纯文本 SLM 在隐空间直接交换视觉信息，绕过离散文本转换造成的信息损失。该方法仅训练约 1.6M 参数，占两个模型总大小的 0.2%。在 CLEVR 和 GQA 视觉问答基准上分别达到 81.0% 和 65.4%，而同等 LoRA 预算下的文本通信基线只有 58.3% 和 61.8%。在 Flickr8k 图像描述任务上比文本通信基线提升 16.5 分。作者还在 TinyLlama-1.1B 与 InternVL2-1B 这对异构架构组合上验证了方法可行性。
Latent-Lens: Visual Perception in Small Language Models Through Latent Communication
Bridging independent vision-language models (VLMs) and small language models (SLMs) in a modular multi-agent system typically forces them to exchange information as discrete text, which creates a lossy bottleneck for spatial and fine-grained details. This constraint is especially acute for SLMs, whose limited capacity makes them more reliant on the quality of the input signal. Recent latent-communication methods exchange states between language models. How a VLM can pass visual information to a separately pretrained text-only SLM though remains unexplored. We propose \textbf{Latent-Lens}, a lightweight codec that bridges the representation gap between a frozen VLM and a frozen SLM directly in latent space, enabling them to communicate purely through hidden states. The codec consists of a domain encoder that projects hidden states from the VLM into the SLM's embedding space, and an attention decoder formalised as a LoRA that teaches the SLM to attend to these visual prefixes, while training only ${\sim}1.6$M parameters (0.2\% of the combined model size). On visual question answering (CLEVR, GQA), Latent-Lens achieves $81.0\pm7.9$\% and $65.4\pm5.4$\%, compared with $58.3\pm5.7$\% and $61.8\pm1.0$\% for text communication with the same LoRA budget (+30.2--69.8pp relative to frozen text communication), while on image captioning (Flickr8k) it improves over the same baseline by +16.5 (+18.8 frozen). We also evaluate Latent-Lens with TinyLlama-1.1B and InternVL2-1B, which use different architectures, vocabularies, and hidden dimensions.