---
id: flashinfer-python
version: "0.6.17"
license: Apache-2.0
license_treatment: permissive
maintenance: active
---

# flashinfer-python — FlashInfer: Kernel Library for LLM Serving

License: permissive · Maintenance: active · Downloads: 5.2M/mo

## What it is and what it does

FlashInfer is a kernel library and generator for high-performance LLM inference on NVIDIA GPUs. It provides unified APIs for attention (including paged/ragged KV-cache, decode, prefill, MLA, cascade, and sparse patterns), matrix multiplication (BF16, FP8, FP4), mixture-of-experts routing, and sampling operations. The library automatically selects the best backend—FlashAttention-2/3, cuDNN, CUTLASS, or TensorRT-LLM—for your hardware and workload.

It is designed for production serving with support for CUDAGraph and torch.compile, low-precision compute (FP8, FP4 quantization), and modern GPU architectures from Turing (SM 7.5) through Blackwell (SM 12.1). The package ships as a pure-Python wheel that compiles or downloads kernels on first use, with optional pre-compiled packages available for faster initialization and offline deployment.

Use it for:

- Accelerate attention computation in LLM inference servers (e.g., vLLM, SGLang, TensorRT-LLM).
- Optimize prefill and decode phases separately for mixed-batch serving scenarios.
- Deploy quantized (FP8/FP4) inference for memory-constrained or cost-sensitive GPU clusters.
- Implement custom attention patterns (sparse, cascade, MLA) without writing CUDA code.
- Reduce latency in speculative decoding and multi-node inference with AllReduce and NVSHMEM support.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

FlashInfer provides optimized GPU kernels for LLM inference operations—attention, GEMM, and mixture-of-experts—with support for multiple GPU architectures and low-precision quantization.

Yes. FlashInfer is worth installing if you are building or deploying LLM inference on NVIDIA GPUs. It is actively maintained, has no known vulnerabilities, carries a permissive license, and is already adopted by major projects (vLLM, SGLang, TensorRT-LLM). Install friction is low. The main constraint is GPU hardware: you must have an NVIDIA GPU with compute capability SM 7.5 or later and CUDA 12.6–13.1.

## Install

pip install flashinfer-python
uv add flashinfer-python
poetry add flashinfer-python

## Installing flashinfer-python

Before you install: Low friction; pure-Python wheel with optional pre-compiled kernel packages. Active maintenance with a release 3 days ago. Requires Python 3.10+, CUDA 12.6–13.1, and NVIDIA GPU with compute capability SM 7.5 or later.

License in practice: Apache-2.0 permissive license allows commercial and private use with minimal restrictions.

Quickstart:

pip install flashinfer-python
import torch
import flashinfer
q = torch.randn(32, 128, device="cuda", dtype=torch.float16)
k = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16)
v = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16)
output = flashinfer.single_decode_with_kv_cache(q, k, v)

Requires NVIDIA GPU with compute capability SM 7.5 or later, CUDA 12.6–13.1, and Python 3.10+.

Verify before relying:

- Whether pre-compiled kernel packages (flashinfer-cubin, flashinfer-jit-cache) are necessary for typical workloads or optional for faster startup.
- Performance gains relative to standard PyTorch attention on specific hardware (e.g., T4, A100, H100).
- Compatibility with non-NVIDIA GPUs or AMD ROCm.

## Package facts

- License: Apache-2.0 (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 5.2M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

gpu kernels llm inference, attention optimization cuda, flashattention implementation, llm serving performance, quantized gemm operations, mixture of experts kernels, kv cache attention, low latency inference, gpu-acceleration, llm-inference, cuda-kernels

[View on SkillFed](https://skillfed.io/packages/flashinfer-python) · [View on PyPI](https://pypi.org/project/flashinfer-python/)