vLLM 推出 TorchTPU 后端，让 Google Cloud TPU 更易用
用 vLLM 的朋友注意：TPU 现在有官方后端了，Pallas kernels 和多机并行都开源，贡献者还能白嫖免费 TPU 算力。
vLLM 与 Google 合作发布 vLLM TorchTPU，这是面向 Google Cloud TPUs 的新后端。借助 vLLM 生态，当前在 vLLM 上运行的模型可以直接部署到 TPU 上。该后端包含开源 Pallas kernels、multi-host parallelism、KV cache offloading 和 disaggregated serving 等能力。团队还为社区贡献者开放了免费 TPU 算力池。作者称目标是让 TPU 像 GPU 一样易于使用。
I’ve worked with Google’s internal AI infrastructure, including TPU, since the TensorFlow 1.0 era. It’s an excellent system, but it was hindered by layers of overlays and complexity that none of us could fully control.
My dream has always been to make TPU as accessible as GPUs. I’ve never been more excited than I am today, because we’ve finally found the path: opening up the knobs and plugging TPU into the community, end to end.
Today we’re releasing vLLM TorchTPU with Google, a new vLLM backend for Google Cloud TPUs. By leveraging vLLM ecosystem, more models that run in vLLM today will be served on TPU. It also brings open-source Pallas kernels, multi-host parallelism, KV cache offloading, and disaggregated serving. We’re even opening a free TPU capacity pool for community contributors.
This only happened because of the amazing Inferact and Google teams. They are some of the sharpest and most generous collaborators I’ve worked with. I’m honored to contribute, and I’m excited to see what the community builds. Let's cook
Blog: https://t.co/YR5M2x9Gvh
