---
id: llmcompressor
version: "0.13.0"
license: Apache 2.0
license_treatment: permissive
maintenance: active
---

# llmcompressor — A library for compressing large language models utilizing the latest techniques and research in the field for both training aware and post training techniques. The library is designed to be flexible and easy to use on top of PyTorch and HuggingFace Transformers, allowing for quick experimentation.

License: permissive · Maintenance: active · Downloads: 221.4K/mo

## What it is and what it does

llmcompressor is a PyTorch-based library for compressing large language models through quantization, pruning, and related techniques. It integrates with Hugging Face models and outputs compressed checkpoints in the compressed-tensors format, which vLLM can load directly for inference. The library supports multiple quantization precisions (int8, fp8, NVFP4, MXFP4, etc.) and algorithms (GPTQ, AWQ, SmoothQuant, AutoRound, REAP), with built-in support for weight-only, weight-activation, KV cache, and attention quantization.

Typical usage involves loading a Hugging Face model, applying a compression recipe (via YAML configuration or Python API), and saving the result for deployment. The library handles distributed training (DDP) and disk offloading to compress very large models on limited hardware. It depends on torch, transformers, datasets, accelerate, and several specialized packages like auto-round and compressed-tensors.

Use it for:

- Reduce model size and memory footprint for single-GPU deployment of large models like Llama or Qwen variants.
- Apply post-training quantization (PTQ) to existing checkpoints without retraining, using calibration data.
- Compress Mixture-of-Experts models by pruning less-relevant experts while maintaining accuracy.
- Prepare quantized models for vLLM inference with guaranteed format compatibility.
- Experiment with different quantization schemes (W4A8, W8A16, NVFP4, etc.) on custom models.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

llmcompressor optimizes large language models for efficient deployment by applying quantization, pruning, and other compression techniques compatible with vLLM inference.

Yes. Active maintenance, no known vulnerabilities, permissive license, and low install friction make it a solid choice for anyone deploying LLMs with vLLM. The substantial dependency footprint (torch, transformers, etc.) is expected for this use case. Start with the step-by-step compression guide in the documentation to select an appropriate quantization scheme for your model and hardware.

## Install

pip install llmcompressor
uv add llmcompressor
poetry add llmcompressor

## Installing llmcompressor

Before you install: Low friction install with a pure-Python wheel. Active maintenance with a recent release (3 days old) and 3678 repository stars. Requires 13 runtime dependencies including torch, transformers, and datasets—a substantial but standard ML stack.

License in practice: Apache 2.0 permissive license allows commercial and private use with minimal restrictions, making it suitable for production deployment scenarios.

Quickstart:

pip install llmcompressor

from llmcompressor.transformers import SparseAutoModelForCausalLM
from transformers import AutoTokenizer

model = SparseAutoModelForCausalLM.from_pretrained(
    "RedHatAI/Llama-2-7b-chat-hf-W4A8-GPTQ"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
inputs = tokenizer("Hello world", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)

Requires Python 3.10+. torch and transformers must be installed; llmcompressor will not function without them. GPU recommended for practical compression workflows.

Verify before relying:

- Whether quantized models maintain accuracy on downstream tasks beyond the calibration set used during compression.
- Performance gains and memory savings for specific model sizes and quantization schemes in your target hardware.
- Compatibility with vLLM versions beyond what the fact sheet documents.

## Package facts

- License: Apache 2.0 (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 221.4K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

llm quantization, model compression pytorch, vllm optimization, weight quantization, language model pruning, activation quantization, model efficiency inference, huggingface model compression, model-optimization, quantization, inference-acceleration

[View on SkillFed](https://skillfed.io/packages/llmcompressor) · [View on PyPI](https://pypi.org/project/llmcompressor/)