StreamEmbed:基于像素差异的增量式截图实时索引系统
StreamEmbed: Real-Time Indexing from Pixels for Screenshot Retrieval
做屏幕记忆检索的朋友看这篇:OCR 太快但丢视觉信息,视觉编码又超两秒。StreamEmbed 靠像素差异跳过重复块、复用缓存,M2 Pro 上快 3.7 倍还不掉点,一年截图只占 0.13GB 内存。
论文提出 StreamEmbed,一种针对屏幕截图流的增量式视觉嵌入系统,用于将屏幕历史构建为可检索的视觉记忆。其核心机制仅由像素差异驱动:空间 token 剪枝跳过与相邻块重复的区域,时序 KV 复用对跨截图未变化的块直接复用缓存的 key 和 value,再由小型学习适配器补偿剪枝和复用带来的损失。
在 ScreenMemEval 基准(10,426 个问题、覆盖两名用户和一个 computer-use agent 共 184 小时的屏幕历史)上,StreamEmbed 在 M2 Pro GPU 上实现最高 3.7 倍于全截图编码的速度且精度不降,同延迟下比 OCR 索引最高提升 6.3 个 Recall@10 点。配套的省内存向量索引以 0.13 GB RAM 存储模拟一年的截图。
OCR 索引速度快但丢失视觉信息,全量视觉编码则超过两秒、无法实时。StreamEmbed 利用了屏幕画面在时间上的高度冗余,证明视觉编码器不必对每张新截图从头计算,为完全本地化运行的视觉助手提供了可行性基础。
StreamEmbed: Real-Time Indexing from Pixels for Screenshot Retrieval
Screen history can serve as searchable visual memory for people and agents, but unlike an offline corpus, it must be encoded and indexed as each screenshot arrives. OCR-based indexing is fast but loses visual information; visual encoding takes over two seconds. Yet screens change incrementally: most pixels repeat within and across screenshots. This raises a question: do visual encoders need to recompute their representations from scratch for every new screenshot? We present StreamEmbed, a system for incremental visual embedding that exploits this redundancy. Guided solely by pixel differences, it combines spatial token pruning, which skips blocks that repeat a neighbor, with temporal KV reuse, which reuses cached keys and values for blocks unchanged across screenshots. A small learned adapter compensates for pruning and reuse to preserve fidelity to full-screenshot embeddings. On ScreenMemEval, our benchmark of 10,426 questions over 184 hours of screen histories from two humans and a computer-use agent, StreamEmbed sets a new accuracy-latency Pareto frontier on an M2 Pro GPU: it encodes up to 3.7x faster than full-screenshot encoding without sacrificing accuracy and improves accuracy over OCR-based indexing by up to 6.3 Recall@10 points at matched latency. A memory-efficient vector index stores a simulated year of screenshots in 0.13 GB of RAM. Together, these results make real-time indexing of screen history in its native visual form practical, laying a foundation for fully local visual assistants.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:StreamEmbed、ScreenMemEval、视觉记忆、RAG,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Dayoon Ko, Yichuan Wang, Soyeong Jeong, Joseph E. Gonzalez, Gunhee Kim, Sewon Min),经 AITOP 多维评分管线结构化提炼。