论文

Spool:让混合架构 VLM 流式视频推理更快更准

Spool: Low-Latency Streaming Inference for Hybrid Vision-Language Models

精选理由

做视频流式推理的可以看看这篇。它不训练任何参数,靠缓存每帧的状态变换来删掉过期帧,首字延迟比 vLLM 基线快 6.6 倍,准确率还涨了 2.3 个百分点。写机器人或实时视频问答的工程师能直接用上这套思路。

论文 Spool(arXiv 2611.23228)针对混合视觉语言模型的滑动窗口视频推理问题提出免训练方案。混合 VLM(如带 Gated DeltaNet 线性递归层的模型)在维护过去两分钟等有限帧窗口时,通常需要把整个窗口重跑一遍模型才能剔除过期帧,延迟开销很大。

正文

核心发现是递归层的状态更新对状态是仿射的、系数依赖输入,因此每帧的影响可表示为一个满足结合律的变换。Spool 缓存逐帧状态变换,窗口推进时直接组合这些变换,从每个递归层状态中精确移除过期帧,无需回放帧通过主干网络,并配了专用数据结构和 GPU kernel 降低缓存与组合开销。

在 5 个开源混合 VLM 和 4 个视频推理基准共 20 组测试中,Spool 全部取得最低 p95 首字延迟,相较两个近期 vLLM 基线中更快者几何平均降低 6.6 倍,准确率平均提升 2.3 个百分点。该方案适用于视觉无障碍助手、机器人等需要快速响应近期视频内容的场景,对做视频流式推理系统优化的工程师有直接参考价值。

原文 · arXiv cs.LG

Spool: Low-Latency Streaming Inference for Hybrid Vision-Language Models

Applications such as visual accessibility agents and robotics require answering questions about recent video content promptly, over a bounded window of past frames (e.g., the last two minutes). Hybrid vision-language models (VLMs), which interleave attention with linear recurrent layers such as Gated DeltaNet (GDN), are attractive for this setting, but maintaining a sliding window is expensive. The reason is that recurrent layers combine all past frames into a shared state, so expiring old frames normally requires re-running the entire window through the model. We observe that the recurrent update is affine in the state with input-dependent coefficients, so the effect of each frame can be captured as a transformation that composes associatively. We present Spool, a training-free framework that caches per-frame state transformations and composes them efficiently as the window advances, removing expired frames from each recurrent layer's state without replaying frames through the backbone. Removal is exact within each layer; as with key-value eviction in attention, expired frames can retain residual influence through the cached inputs computed by earlier layers. We design a dedicated data structure and GPU kernels to achieve low caching and composition overhead. Across five open-source hybrid VLMs and four video reasoning benchmarks, Spool achieves the lowest p95 time-to-first-token in all 20 model-benchmark tests, reducing it by a geometric mean of $6.6\times$ over the faster of two recent vLLM baselines, while improving accuracy by 2.3 percentage points on average.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Spool、VLM、Gated DeltaNet、vLLM,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Rui Ning, Om Chabra, Hari Balakrishnan),经 AITOP 多维评分管线结构化提炼。