flashinfer-python
FlashInfer: Kernel Library for LLM Serving
Decision gist · record as of 2026-08-14
Yes. FlashInfer is worth installing if you are building or deploying LLM inference on NVIDIA GPUs. It is actively maintained, has no known vulnerabilities, carries a permissive license, and is already adopted by major projects (vLLM, SGLang, TensorRT-LLM). Install friction is low. The main constraint is GPU hardware: you must have an NVIDIA GPU with compute capability SM 7.5 or later and CUDA 12.6–13.1.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires NVIDIA GPU with compute capability SM 7.5 or later, CUDA 12.6–13.1, and Python 3.10+.
- Low friction; pure-Python wheel with optional pre-compiled kernel packages.
- Active maintenance with a release 3 days ago.
License · maintenance · safety
Apache-2.0 (permissive) — Apache-2.0 permissive license allows commercial and private use with minimal restrictions.
last release 2026-08-11 (3 days) · last repo commit 2026-08-14 · 6,159 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 5,180,758 downloads/mo, #2,147 on PyPI
Alternatives
Verify before relying
pip install flashinfer-python
import torch
import flashinfer
q = torch.randn(32, 128, device="cuda", dtype=torch.float16)
k = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16)
v = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16)
output = flashinfer.single_decode_with_kv_cache(q, k, v)- Whether pre-compiled kernel packages (flashinfer-cubin, flashinfer-jit-cache) are necessary for typical workloads or optional for faster startup.
- Performance gains relative to standard PyTorch attention on specific hardware (e.g., T4, A100, H100).
- Compatibility with non-NVIDIA GPUs or AMD ROCm.
What it is and what it does
FlashInfer is a kernel library and generator for high-performance LLM inference on NVIDIA GPUs. It provides unified APIs for attention (including paged/ragged KV-cache, decode, prefill, MLA, cascade, and sparse patterns), matrix multiplication (BF16, FP8, FP4), mixture-of-experts routing, and sampling operations. The library automatically selects the best backend—FlashAttention-2/3, cuDNN, CUTLASS, or TensorRT-LLM—for your hardware and workload.
It is designed for production serving with support for CUDAGraph and torch.compile, low-precision compute (FP8, FP4 quantization), and modern GPU architectures from Turing (SM 7.5) through Blackwell (SM 12.1). The package ships as a pure-Python wheel that compiles or downloads kernels on first use, with optional pre-compiled packages available for faster initialization and offline deployment.
Use it for
- Accelerate attention computation in LLM inference servers (e.g., vLLM, SGLang, TensorRT-LLM).
- Optimize prefill and decode phases separately for mixed-batch serving scenarios.
- Deploy quantized (FP8/FP4) inference for memory-constrained or cost-sensitive GPU clusters.
- Implement custom attention patterns (sparse, cascade, MLA) without writing CUDA code.
- Reduce latency in speculative decoding and multi-node inference with AllReduce and NVSHMEM support.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
FlashInfer is worth installing if you are building or deploying LLM inference on NVIDIA GPUs. It is actively maintained, has no known vulnerabilities, carries a permissive license, and is already adopted by major projects (vLLM, SGLang, TensorRT-LLM). Install friction is low. The main constraint is GPU hardware: you must have an NVIDIA GPU with compute capability SM 7.5 or later and CUDA 12.6–13.1.
Install
flashinfer-python on PyPI
Before you install
Low friction; pure-Python wheel with optional pre-compiled kernel packages. Active maintenance with a release 3 days ago. Requires Python 3.10+, CUDA 12.6–13.1, and NVIDIA GPU with compute capability SM 7.5 or later.
Requires NVIDIA GPU with compute capability SM 7.5 or later, CUDA 12.6–13.1, and Python 3.10+.
License in practice
Apache-2.0 permissive license allows commercial and private use with minimal restrictions.
Quickstart
pip install flashinfer-python
import torch
import flashinfer
q = torch.randn(32, 128, device="cuda", dtype=torch.float16)
k = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16)
v = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16)
output = flashinfer.single_decode_with_kv_cache(q, k, v)Verify before relying
- Whether pre-compiled kernel packages (flashinfer-cubin, flashinfer-jit-cache) are necessary for typical workloads or optional for faster startup.
- Performance gains relative to standard PyTorch attention on specific hardware (e.g., T4, A100, H100).
- Compatibility with non-NVIDIA GPUs or AMD ROCm.
Package facts
| License | Apache-2.0 permissive |
| Python support | Supports the current Python release <4.0,>=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 16 packagesapache-tvm-fficlickcuda-pythoncuda-tileeinopsnccl4pyninjanumpynvidia-cudnn-frontendnvidia-cutlass-dslnvidia-ml-pypackagingrequeststabulatetorchtqdm |
| Maintenance | Actively maintained 3 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 5,180,758 / month, #2,147 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
Evidence: flashinfer_python-0.6.17-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “llm serving performance”
- flashinfer-pythonFlashInfer provides optimized GPU kernels for LLM inference…
- fastokensfastokens is a high-performance BPE tokenizer for large language…
- datarobot-genaiA toolkit for building and deploying AI agents on DataRobot,…
Give your agent the search over MCP, or paste the wish link into any chat.
More Artificial Intelligence packages
LiteLLM provides a unified Python interface to call 100+ LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Azure, and others) using OpenAI-compatible API format, available as both a Python SDK and a self-hosted AI Gateway proxy server.
Install it if you need to work with multiple LLM providers or want to centralize LLM routing in your organization.
Client library and CLI tool for downloading, uploading, and managing models, datasets, and repositories on the Hugging Face Hub platform.
Install it if you work with Hugging Face Hub models or datasets.
LangChain provides a framework for building agents and LLM-powered applications by composing language models, tools, and memory through a unified API that abstracts over multiple model providers.
hf-xet provides chunk-based deduplication and efficient file transfer for the Hugging Face Hub, enabling faster uploads and downloads of large files with local disk caching.
Tokenizers converts raw text into token sequences for NLP models, with support for training custom vocabularies and using pre-built tokenizers (BPE, WordPiece) optimized for speed via Rust.
Transformers provides a unified framework for loading, fine-tuning, and running state-of-the-art pretrained models across text, vision, audio, video, and multimodal tasks using PyTorch, JAX, or TensorFlow.
Install it if you need to run or train any transformer-based model for NLP, vision, audio, or multimodal tasks.
See also flashinfer-cubin · sageattention · fa3-fwd · tilelang · sgl-deep-gemm · sgl-kernel · sglang-kernel · tokenspeed-mla · nvidia-cudnn-frontend · humming-kernels