tpu-inference

What it is and what it does

tpu-inference is a vLLM plugin that brings unified inference serving to Google TPUs, bridging PyTorch and JAX model ecosystems under a single lowering path. It allows developers to run PyTorch model definitions natively on TPU hardware without code changes, while also extending native JAX support, all while maintaining vLLM's standard user interface and telemetry. The package targets TPU generations v3 through v7x, with v5e, v6e, and v7x as the recommended targets.

The plugin is designed for production LLM serving workloads. It includes support for core inference features like async scheduling, chunked prefill, KV cache offload, prefix caching, and multimodal inputs. The fact sheet documents tested models including Gemma, Llama, and Qwen families, though some advanced features remain experimental or untested. Installation requires compatible TPU hardware and brings in dependencies across the JAX, PyTorch, and Google Cloud ecosystems.

Use it for:

• Serve open-source LLMs like Llama 3.1/3.3 or Gemma on TPU infrastructure for production inference workloads.
• Run PyTorch-defined models on TPU hardware without rewriting model code, leveraging TPU performance.
• Deploy multimodal models (vision-language) on TPUs with tested support for Gemma-4 and Qwen VL variants.
• Build cost-optimized inference services using TPU's price-to-performance characteristics compared to GPU alternatives.
• Develop JAX-native inference pipelines with unified backend support alongside PyTorch workloads.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

tpu-inference is a hardware plugin for vLLM that enables high-performance serving of large language models on Google TPUs, supporting both PyTorch and JAX model definitions through a unified backend.

Yes, if you have access to Google TPU hardware and need to serve large language models at scale. The package is actively maintained, permissively licensed, and offers low install friction. However, it is strictly tied to TPU infrastructure—it cannot run on CPU or GPU systems. Verify that your target models appear in the tested support matrix and that required features are marked passing rather than experimental or untested.

Install

tpu-inference on PyPI

pip install tpu-inference

uv add tpu-inference

poetry add tpu-inference

Installing tpu-inference

Before you install

Low install friction with a pure-Python wheel distribution. Active maintenance with a recent release 14 days ago. Depends on 24 runtime packages including JAX, PyTorch ecosystem libraries, and Google Cloud integrations, which may require significant disk space and compatible system setup.

License in practice

Licensed under Apache Software License (permissive), allowing commercial and private use with minimal restrictions.

Quickstart

pip install tpu-inference

from tpu_inference import TPUInference
# Requires vLLM and TPU hardware to instantiate and serve models

Requires Google TPU hardware (v3, v4, v5p, v5e, v6e, or v7x) and a compatible TPU environment; cannot run on CPU or GPU systems.

Verify before relying

• Whether tpu-inference can be installed and imported without TPU hardware present for development/testing purposes.
• Exact performance characteristics and throughput improvements compared to other TPU serving solutions.
• Whether all 24 runtime dependencies are strictly required or if some are optional for specific use cases.

Package facts

Evidence: tpu_inference-0.26.0-py3-none-any.whl

Tags

More Artificial Intelligence packages