humming-kernels

What it is and what it does

Humming is a lightweight, JIT-compiled GEMM kernel library optimized for quantized inference on NVIDIA GPUs. It provides high-performance matrix multiplication kernels that support a wide range of quantization formats—from FP16 and BF16 down to FP4 and INT4 weights—paired with various activation types (FP16, BF16, FP8, INT8, INT4). The library handles both dense matrix operations and mixture-of-experts (MoE) patterns, making it suitable for deploying quantized large language models and other inference workloads.

The package is designed to be minimal and self-contained, requiring only PyTorch and NVCC as core dependencies, with a compact footprint under 100KB. It abstracts away kernel tuning through a HummingLayer interface that automatically selects appropriate kernels for your hardware and quantization configuration. You load unquantized weights, transform them into Humming's internal format, and then run inference through the layer—the library handles the low-level kernel dispatch and optimization.

Use it for:

• Deploy quantized LLMs with mixed-bit weight formats (INT4, INT6, INT8) on NVIDIA GPUs for reduced memory and latency.
• Run inference with FP8 or FP4 activations and weights on newer GPUs (SM89+) for extreme compression.
• Accelerate MoE model inference by using Humming's specialized MoE GEMM kernels instead of generic matrix operations.
• Benchmark quantization strategies across different bit-widths and scale types without writing custom CUDA code.
• Integrate quantized inference into production systems where minimal dependencies and small package size are constraints.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Humming is a JIT-compiled GEMM kernel library for quantized matrix multiplication on NVIDIA GPUs, supporting mixed quantization formats (FP16, BF16, FP8, FP4, INT8, INT4) for both dense and MoE inference workloads.

Yes, if you need quantized GEMM kernels for inference on NVIDIA GPUs and have the required hardware (SM75+) and Python environment (≥3.10). The package is actively maintained, has no known vulnerabilities, and offers broad quantization format support. However, verify the license terms before use, and confirm that your CUDA and PyTorch versions are compatible—the fact sheet does not specify exact version constraints beyond the architecture requirement.

Install

humming-kernels on PyPI

pip install humming-kernels

uv add humming-kernels

poetry add humming-kernels

Installing humming-kernels

Before you install

Medium install friction due to 9 runtime dependencies including torch, triton, and cuda-bindings. Package is actively maintained with recent releases, but availability is limited to manylinux wheels for x86_64 and aarch64 architectures.

License in practice

License status is unclear—no SPDX identifier or raw license text is available. Verify the project's actual license terms before use in proprietary or redistributed code.

Quickstart

pip install humming-kernels

import torch
from humming.layer import HummingLayer

layer = HummingLayer(
    shape_n=8192,
    shape_k=8192,
    weight_config={"dtype": "int6"},
    torch_dtype=torch.float16,
).cuda()

weight = torch.randn((8192, 8192), dtype=torch.float16, device="cuda:0")
inputs = torch.randn((128, 8192), dtype=torch.float16, device="cuda:0")
layer.load_from_unquantized(weight)
layer.transform()
output = layer(inputs)

Requires NVIDIA GPU with SM75+ architecture (Turing or newer); NVCC compiler and CUDA toolkit must be available; Python >=3.10.

Verify before relying

• Whether the package's license permits commercial use and redistribution—license_treatment is unclear.
• Performance benchmarks and throughput claims relative to other quantized GEMM libraries.
• Exact CUDA and PyTorch version compatibility constraints beyond the stated SM architecture requirements.

Package facts

Evidence: humming_kernels-0.1.12-py3-none-manylinux_2_28_aarch64.whl; humming_kernels-0.1.12-py3-none-manylinux_2_28_x86_64.whl

Tags

More Artificial Intelligence packages

Further reading