{"categories":[{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/11"}],"enrichment":{"capability":"tpu-inference is a hardware plugin for vLLM that enables high-performance serving of large language models on Google TPUs, supporting both PyTorch and JAX model definitions through a unified backend.","skillfed_tags":["tpu-serving","llm-inference","jax-pytorch-unified"],"use_cases":["Serve open-source LLMs like Llama 3.1/3.3 or Gemma on TPU infrastructure for production inference workloads.","Run PyTorch-defined models on TPU hardware without rewriting model code, leveraging TPU performance.","Deploy multimodal models (vision-language) on TPUs with tested support for Gemma-4 and Qwen VL variants.","Build cost-optimized inference services using TPU's price-to-performance characteristics compared to GPU alternatives.","Develop JAX-native inference pipelines with unified backend support alongside PyTorch workloads."],"what_it_does":"tpu-inference is a vLLM plugin that brings unified inference serving to Google TPUs, bridging PyTorch and JAX model ecosystems under a single lowering path. It allows developers to run PyTorch model definitions natively on TPU hardware without code changes, while also extending native JAX support, all while maintaining vLLM's standard user interface and telemetry. The package targets TPU generations v3 through v7x, with v5e, v6e, and v7x as the recommended targets.\n\nThe plugin is designed for production LLM serving workloads. It includes support for core inference features like async scheduling, chunked prefill, KV cache offload, prefix caching, and multimodal inputs. The fact sheet documents tested models including Gemma, Llama, and Qwen families, though some advanced features remain experimental or untested. Installation requires compatible TPU hardware and brings in dependencies across the JAX, PyTorch, and Google Cloud ecosystems.","worth_installing":"Yes, if you have access to Google TPU hardware and need to serve large language models at scale. The package is actively maintained, permissively licensed, and offers low install friction. However, it is strictly tied to TPU infrastructure\u2014it cannot run on CPU or GPU systems. Verify that your target models appear in the tested support matrix and that required features are marked passing rather than experimental or untested."},"id":"tpu-inference","links":{"html":"https://skillfed.io/packages/tpu-inference","md":"https://skillfed.io/packages/tpu-inference.md","pypi":"https://pypi.org/project/tpu-inference/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-07-31","license_spdx":null,"license_treatment":"permissive","name":"tpu-inference","python_support":"supports_current","summary":null},"popularity":{"monthly_downloads":79219,"position":14373,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.26.0"}