llmcompressor

What it is and what it does

llmcompressor is a PyTorch-based library for compressing large language models through quantization, pruning, and related techniques. It integrates with Hugging Face models and outputs compressed checkpoints in the compressed-tensors format, which vLLM can load directly for inference. The library supports multiple quantization precisions (int8, fp8, NVFP4, MXFP4, etc.) and algorithms (GPTQ, AWQ, SmoothQuant, AutoRound, REAP), with built-in support for weight-only, weight-activation, KV cache, and attention quantization.

Typical usage involves loading a Hugging Face model, applying a compression recipe (via YAML configuration or Python API), and saving the result for deployment. The library handles distributed training (DDP) and disk offloading to compress very large models on limited hardware. It depends on torch, transformers, datasets, accelerate, and several specialized packages like auto-round and compressed-tensors.

Use it for:

• Reduce model size and memory footprint for single-GPU deployment of large models like Llama or Qwen variants.
• Apply post-training quantization (PTQ) to existing checkpoints without retraining, using calibration data.
• Compress Mixture-of-Experts models by pruning less-relevant experts while maintaining accuracy.
• Prepare quantized models for vLLM inference with guaranteed format compatibility.
• Experiment with different quantization schemes (W4A8, W8A16, NVFP4, etc.) on custom models.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

llmcompressor optimizes large language models for efficient deployment by applying quantization, pruning, and other compression techniques compatible with vLLM inference.

Yes. Active maintenance, no known vulnerabilities, permissive license, and low install friction make it a solid choice for anyone deploying LLMs with vLLM. The substantial dependency footprint (torch, transformers, etc.) is expected for this use case. Start with the step-by-step compression guide in the documentation to select an appropriate quantization scheme for your model and hardware.

Install

llmcompressor on PyPI

pip install llmcompressor

uv add llmcompressor

poetry add llmcompressor

Installing llmcompressor

Before you install

Low friction install with a pure-Python wheel. Active maintenance with a recent release (3 days old) and 3678 repository stars. Requires 13 runtime dependencies including torch, transformers, and datasets—a substantial but standard ML stack.

License in practice

Apache 2.0 permissive license allows commercial and private use with minimal restrictions, making it suitable for production deployment scenarios.

Quickstart

pip install llmcompressor

from llmcompressor.transformers import SparseAutoModelForCausalLM
from transformers import AutoTokenizer

model = SparseAutoModelForCausalLM.from_pretrained(
    "RedHatAI/Llama-2-7b-chat-hf-W4A8-GPTQ"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
inputs = tokenizer("Hello world", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)

Requires Python 3.10+. torch and transformers must be installed; llmcompressor will not function without them. GPU recommended for practical compression workflows.

Verify before relying

• Whether quantized models maintain accuracy on downstream tasks beyond the calibration set used during compression.
• Performance gains and memory savings for specific model sizes and quantization schemes in your target hardware.
• Compatibility with vLLM versions beyond what the fact sheet documents.

Package facts

Evidence: llmcompressor-0.13.0-py3-none-any.whl

Keywords: llmcompressor, llms, large language models, transformers, pytorch, huggingface, compressors, compression, quantization, pruning, sparsity, optimization, model optimization, model compression

Tags

More Software Development packages

Further reading