--- id: tpu-inference version: "0.26.0" license: unclear license_treatment: permissive maintenance: active --- # tpu-inference License: permissive · Maintenance: active · Downloads: 79.2K/mo ## What it is and what it does tpu-inference is a vLLM plugin that brings unified inference serving to Google TPUs, bridging PyTorch and JAX model ecosystems under a single lowering path. It allows developers to run PyTorch model definitions natively on TPU hardware without code changes, while also extending native JAX support, all while maintaining vLLM's standard user interface and telemetry. The package targets TPU generations v3 through v7x, with v5e, v6e, and v7x as the recommended targets. The plugin is designed for production LLM serving workloads. It includes support for core inference features like async scheduling, chunked prefill, KV cache offload, prefix caching, and multimodal inputs. The fact sheet documents tested models including Gemma, Llama, and Qwen families, though some advanced features remain experimental or untested. Installation requires compatible TPU hardware and brings in dependencies across the JAX, PyTorch, and Google Cloud ecosystems. Use it for: - Serve open-source LLMs like Llama 3.1/3.3 or Gemma on TPU infrastructure for production inference workloads. - Run PyTorch-defined models on TPU hardware without rewriting model code, leveraging TPU performance. - Deploy multimodal models (vision-language) on TPUs with tested support for Gemma-4 and Qwen VL variants. - Build cost-optimized inference services using TPU's price-to-performance characteristics compared to GPU alternatives. - Develop JAX-native inference pipelines with unified backend support alongside PyTorch workloads. ## Worth the install? AI-flagged interpretation of the facts above — verify before relying. tpu-inference is a hardware plugin for vLLM that enables high-performance serving of large language models on Google TPUs, supporting both PyTorch and JAX model definitions through a unified backend. Yes, if you have access to Google TPU hardware and need to serve large language models at scale. The package is actively maintained, permissively licensed, and offers low install friction. However, it is strictly tied to TPU infrastructure—it cannot run on CPU or GPU systems. Verify that your target models appear in the tested support matrix and that required features are marked passing rather than experimental or untested. ## Install pip install tpu-inference uv add tpu-inference poetry add tpu-inference ## Installing tpu-inference Before you install: Low install friction with a pure-Python wheel distribution. Active maintenance with a recent release 14 days ago. Depends on 24 runtime packages including JAX, PyTorch ecosystem libraries, and Google Cloud integrations, which may require significant disk space and compatible system setup. License in practice: Licensed under Apache Software License (permissive), allowing commercial and private use with minimal restrictions. Quickstart: pip install tpu-inference from tpu_inference import TPUInference # Requires vLLM and TPU hardware to instantiate and serve models Requires Google TPU hardware (v3, v4, v5p, v5e, v6e, or v7x) and a compatible TPU environment; cannot run on CPU or GPU systems. Verify before relying: - Whether tpu-inference can be installed and imported without TPU hardware present for development/testing purposes. - Exact performance characteristics and throughput improvements compared to other TPU serving solutions. - Whether all 24 runtime dependencies are strictly required or if some are optional for specific use cases. ## Package facts - License: not declared (permissive) - Python support: supports_current - Install friction: low - Maintenance: active - Downloads: 79.2K/month (top 15,000 on PyPI) - Known vulnerabilities: none known ## Tags TPU inference serving, vLLM TPU backend, large language model TPU deployment, JAX PyTorch TPU unified, TPU LLM inference plugin, vLLM hardware acceleration TPU, TPU model serving framework, tpu-serving, llm-inference, jax-pytorch-unified [View on SkillFed](https://skillfed.io/packages/tpu-inference) · [View on PyPI](https://pypi.org/project/tpu-inference/)