推测解码开源工具让大模型生成文本更快
精选理由
用小模型猜 token、大模型验证，一次能吐好几个 token，代码开源了，vLLM 和 SGLang 都能跑，推理党可以试试。
该工具采用推测解码思路：用一个小模型一次猜测多个后续 token，主模型再并行验证这些猜测。猜对时一次前向传播就能输出多个 token，生成速度明显提升。代码已开源，支持在 vLLM 和 SGLang 上直接运行。
原文 · Ate-a-Pi
This will make language models generate text much faster.
Normally, a model generates one token at a time.
But with this tool, a smaller helper model guesses several upcoming tokens, and the main model checks those guesses together.
When the guesses are right, you get several tokens in one pass.
They made the code available, and you can run this with vLLM and SGLang.
