LlamaIndex 将 OCRia-1 在单张 B200 上提速 10 倍
LlamaIndex 把 OCRia-1 在一张 B200 上跑出 2800 tok/s，17 个小优化加量化和投机解码，做推理优化的人可以照抄作业。
LlamaIndex 的文档解析模型 OCRia-1 在单张 B200 上从 285 tok/s 提升到 2800 tok/s，速度提高约 10 倍。团队用量化与 dflash2 投机解码同时降低显存占用并提高吞吐。另有 17 项小优化，包括 replaySSM、kernel fusion 和 vLLM 改动，每项带来 2–5% 的提升，累积效果显著。部分优化计划回馈开源 vLLM 项目，LlamaParse 的整体延迟也会继续下降。
OCRia-1 is one of the fastest models ever for doc parsing. We sped up OCRia-1 by 10x on a single B200 (285 tok/s --> 2800 tok/s). there are a ton of useful learnings in this extremely detailed blog post for anyone looking to optimize models! * we made use of established practices, like quantization and spec decoding (dflash2), to simultaneously reduce memory usage and increase tokens per second * we made 17 other optimizations - replaySSM, kernel fusion, vLLM changes, and more. Huge shoutout to our research team. They've made it possible for us to serve billions of pages at scale. We've continuing to make massive improvements to bring down latency across the board in LlamaParse. Check it out: llamaindex.ai/blog/the-infer… Your browser does not support the video tag. 🔗 View on Twitter LlamaIndex 🦙 @llama_index We got OCRia-1 generating tokens nearly 10× faster on a single B200. Getting there meant making much better use of the same GPU. We tuned how OCRia-1 runs for the demands of document parsing, testing every change for both speed and accuracy. Some improvements were only 2–5%. Seventeen smaller optimizations later, “just a few percent” starts adding up. Our infrastructure team wrote up what worked, how they tested the changes, and what they learned about making document models run faster. We also plan to contribute some of these optimizations back to the open-source vLLM project. Read the full breakdown → llamaindex.ai/blog/the-infer… Watch the video below to see OCRia-1’s token generation before and after the optimizations. Your browser does not support the video tag. 🔗 View on Twitter 🔗 View Quoted Tweet 💬 6 🔄 1 ❤️ 9 👀 1075 📊 6 ⚡
- LlamaIndex16:49原文
