flashinfer-python

What it is and what it does

FlashInfer is a kernel library and generator for high-performance LLM inference on NVIDIA GPUs. It provides unified APIs for attention (including paged/ragged KV-cache, decode, prefill, MLA, cascade, and sparse patterns), matrix multiplication (BF16, FP8, FP4), mixture-of-experts routing, and sampling operations. The library automatically selects the best backend—FlashAttention-2/3, cuDNN, CUTLASS, or TensorRT-LLM—for your hardware and workload.

It is designed for production serving with support for CUDAGraph and torch.compile, low-precision compute (FP8, FP4 quantization), and modern GPU architectures from Turing (SM 7.5) through Blackwell (SM 12.1). The package ships as a pure-Python wheel that compiles or downloads kernels on first use, with optional pre-compiled packages available for faster initialization and offline deployment.

Use it for:

• Accelerate attention computation in LLM inference servers (e.g., vLLM, SGLang, TensorRT-LLM).
• Optimize prefill and decode phases separately for mixed-batch serving scenarios.
• Deploy quantized (FP8/FP4) inference for memory-constrained or cost-sensitive GPU clusters.
• Implement custom attention patterns (sparse, cascade, MLA) without writing CUDA code.
• Reduce latency in speculative decoding and multi-node inference with AllReduce and NVSHMEM support.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

FlashInfer provides optimized GPU kernels for LLM inference operations—attention, GEMM, and mixture-of-experts—with support for multiple GPU architectures and low-precision quantization.

Yes. FlashInfer is worth installing if you are building or deploying LLM inference on NVIDIA GPUs. It is actively maintained, has no known vulnerabilities, carries a permissive license, and is already adopted by major projects (vLLM, SGLang, TensorRT-LLM). Install friction is low. The main constraint is GPU hardware: you must have an NVIDIA GPU with compute capability SM 7.5 or later and CUDA 12.6–13.1.

Install

flashinfer-python on PyPI

pip install flashinfer-python

uv add flashinfer-python

poetry add flashinfer-python

Installing flashinfer-python

Before you install

Low friction; pure-Python wheel with optional pre-compiled kernel packages. Active maintenance with a release 3 days ago. Requires Python 3.10+, CUDA 12.6–13.1, and NVIDIA GPU with compute capability SM 7.5 or later.

License in practice

Apache-2.0 permissive license allows commercial and private use with minimal restrictions.

Quickstart

pip install flashinfer-python
import torch
import flashinfer
q = torch.randn(32, 128, device="cuda", dtype=torch.float16)
k = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16)
v = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16)
output = flashinfer.single_decode_with_kv_cache(q, k, v)

Requires NVIDIA GPU with compute capability SM 7.5 or later, CUDA 12.6–13.1, and Python 3.10+.

Verify before relying

• Whether pre-compiled kernel packages (flashinfer-cubin, flashinfer-jit-cache) are necessary for typical workloads or optional for faster startup.
• Performance gains relative to standard PyTorch attention on specific hardware (e.g., T4, A100, H100).
• Compatibility with non-NVIDIA GPUs or AMD ROCm.

Package facts

Evidence: flashinfer_python-0.6.17-py3-none-any.whl

Tags

More Artificial Intelligence packages

Further reading