sglang-kernel

What it is and what it does

sglang-kernel is a library of hand-optimized CUDA kernels designed to accelerate inference for large language models and vision-language models. It provides custom compute primitives that integrate with PyTorch, allowing LLM inference engines to execute critical operations more efficiently on NVIDIA GPUs.

The package is distributed as pre-compiled wheels for x86_64 and aarch64 architectures, with no runtime Python dependencies. It requires torch == 2.13.0 and Python ≥3.10, and is intended to be used as a low-level acceleration layer within larger inference frameworks rather than as a standalone application. The source code lives in the sglang project repository.

Use it for:

• Accelerate batch matrix multiplication and attention operations in LLM inference engines
• Optimize vision-language model inference by providing specialized CUDA kernels for image and token processing
• Reduce latency and increase throughput in production LLM serving systems that depend on custom kernel performance
• Integrate with PyTorch-based inference frameworks that need fine-grained control over GPU compute primitives

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

sglang-kernel provides optimized CUDA compute primitives for LLM inference engines, enabling efficient inference for large language models and vision-language models through custom kernel operations.

Yes, if you are building or deploying an LLM inference engine that benefits from custom CUDA kernels and you have torch == 2.13.0 and Python ≥3.10 available. The active maintenance, permissive license, and high repository engagement suggest solid backing. Install friction is moderate due to compiled wheels, but no runtime dependencies simplify integration. Not suitable as a standalone package; it is a dependency for inference frameworks.

Install

sglang-kernel on PyPI

pip install sglang-kernel

uv add sglang-kernel

poetry add sglang-kernel

Installing sglang-kernel

Before you install

Medium install friction due to compiled wheel distribution (aarch64 and x86_64 variants). Active maintenance with recent release 8 days ago and 31807 repository stars. Requires torch == 2.13.0 and Python ≥3.10.

License in practice

Apache License 2.0 (permissive) allows commercial use, modification, and distribution with minimal restrictions. You must include license and copyright notices and document changes.

Quickstart

pip install sglang-kernel
# Import and use within inference framework that depends on sglang-kernel

Requires torch == 2.13.0 and Python ≥3.10. NVIDIA CUDA GPU required. Known segmentation fault with CUDA 12.6; update ptxas to 12.8.

Verify before relying

• Whether pre-built wheels cover all target architectures and CUDA versions needed for your deployment
• Performance gains relative to standard PyTorch operations for your specific model and batch sizes
• Exact kernel operations exposed and their integration patterns within inference frameworks

Package facts

Evidence: sglang_kernel-0.4.6.post1-cp310-abi3-manylinux2014_aarch64.whl; sglang_kernel-0.4.6.post1-cp310-abi3-manylinux2014_x86_64.whl

Tags

More Artificial Intelligence packages

Further reading