sgl-deep-gemm

What it is and what it does

sgl-deep-gemm is an SGLang-optimized fork of DeepGemm that provides GPU-accelerated matrix multiplication kernels for neural network inference and training workloads. It wraps custom GEMM implementations via apache-tvm-ffi to enable ABI-compatible wheels that work across different Python versions without recompilation, simplifying deployment compared to building from source.

The package is designed for use within SGLang's inference framework and targets NVIDIA GPUs on Linux (x86_64 and aarch64). Installation is straightforward via pip, but it requires Python 3.10 or later and a CUDA-capable GPU. The underlying kernels are optimized for specific tensor operations common in large language model inference.

Use it for:

• Accelerate matrix multiplication operations in SGLang-based LLM inference pipelines.
• Deploy optimized GEMM kernels across multiple Python versions without rebuilding wheels.
• Improve throughput for batch matrix operations on NVIDIA GPUs in production systems.
• Integrate custom GPU kernels into neural network frameworks via TVM's FFI layer.
• Support both x86_64 and aarch64 GPU deployments with a single wheel distribution.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

sgl-deep-gemm provides optimized GEMM (matrix multiplication) kernels for GPU acceleration, packaged as a PyPI-installable wheel with ABI compatibility across Python versions via apache-tvm-ffi.

Yes, if you are using SGLang for LLM inference on NVIDIA GPUs and need optimized matrix multiplication. The active maintenance, permissive license, and ABI-compatible wheels make it low-friction to adopt. No, if you are not in the SGLang ecosystem or lack a compatible GPU and Linux platform (x86_64 or aarch64).

Install

sgl-deep-gemm on PyPI

pip install sgl-deep-gemm

uv add sgl-deep-gemm

poetry add sgl-deep-gemm

Installing sgl-deep-gemm

Before you install

Medium install friction due to platform-specific wheels (manylinux2014 x86_64 and aarch64 only) and a compiled dependency on apache-tvm-ffi. Active maintenance with a recent release (9 days old) and steady repository activity.

License in practice

Apache-2.0 permissive license allows commercial and private use with minimal restrictions, making it suitable for most deployment scenarios.

Quickstart

pip install sgl-deep-gemm

import sgl_deep_gemm
# Use optimized GEMM kernels via apache-tvm-ffi bindings

Requires Python >=3.10 and NVIDIA CUDA-capable GPU; wheels available only for manylinux2014 x86_64 and aarch64 architectures.

Verify before relying

• What specific GEMM operations and tensor shapes the kernels optimize for.
• Performance benchmarks compared to standard CUDA GEMM implementations.
• Whether apache-tvm-ffi is automatically installed or requires separate setup.
• Compatibility with specific CUDA and cuDNN versions.

Package facts

Evidence: sgl_deep_gemm-0.1.5.post2-py3-none-manylinux2014_aarch64.whl; sgl_deep_gemm-0.1.5.post2-py3-none-manylinux2014_x86_64.whl

Tags

More Artificial Intelligence packages

Further reading