YC Paper Club 推理专题：训练 1 FLOP 对应每年 6 FLOP 推理开销
YC 拉来 Together AI、Ollama、SGLang 的人讲透推理栈，从 token 请求到 GPU 全链路，还解释了同一开源模型为啥各家跑得不一样。
Y Combinator 本期 Paper Club 聚焦推理环节。核心数据是每花 1 FLOP 训练模型，每年约需 6 FLOP 用于服务，过去十年推理优化催生了超 2200 亿美元市值。内容覆盖一个 token 从 API 请求到 GPU 再返回的完整链路，包括路由、缓存、调度、内核与内存。嘉宾包括 Together AI 的 Dan Fu、Ollama 的 Jeff Morgan、Luminal 的 Joe Fioti，以及讲解 SGLang 内存方案的 Liangsheng Yin。视频还解释了为什么同一个开源模型在不同服务商手里表现不同。
This week’s Paper Club is all about inference. For every FLOP spent training a model, roughly six go to serving it each year — and getting that part right has produced over $220 billion in market cap in the last decade. But inference is much more than calling model.forward. Every token has to move through a complicated stack of routing, caching, scheduling, kernels, memory, and hardware, all while balancing speed, cost, and quality. We brought together researchers and builders from @togethercompute , @ollama , @luminal_ai , and @sgl_project & @radixark to take us through that stack from end to end. We’ll see how a token gets from an API request to a GPU and back, why the same open model can perform differently depending on who serves it, and how new approaches to compilers and caching are making inference faster and more efficient. 00:00 - Why inference matters more than training 08:24 - Three kinds of inference innovation 12:14 - Dan Fu ( @realDanFu ): the lifecycle of an inference request 42:18 - Jeff Morgan ( @jmorgan ): not all open model tokens are the same 1:04:31 - Joe Fioti ( @joefioti ): compiling models with search, not heuristics 1:30:30 - Liangsheng Yin ( @lsyincs ): SGLang and memory for the agent era Your browser does not support the video tag. 🔗 View on Twitter 💬 20 🔄 15 ❤️ 69 👀 11333 📊 30 ⚡
