auto-gptq

What it is and what it does

AutoGPTQ is a quantization library that compresses large language models to 4-bit or lower precision using the GPTQ weight-only quantization algorithm. It integrates with transformers and provides APIs to quantize models, save them in compressed form, and load them for inference on GPUs. The library is designed to reduce memory footprint and accelerate inference speed—the documentation shows examples where quantized models achieve 25.53 tokens/second on A100 GPUs compared to 18.87 tokens/second for full-precision equivalents.

The package depends on a substantial stack: torch, transformers, peft, accelerate, datasets, and several utility libraries. It supports CUDA 11.8, 12.1, and ROCm 5.7 via pre-built wheels for Python 3.8–3.11 on Linux and Windows. However, the repository is now archived and has not been updated since March 2024, meaning no new features, bug fixes, or compatibility patches are expected. Users relying on this package should be prepared to maintain it themselves or migrate to actively maintained alternatives if breaking changes occur in upstream dependencies.

Use it for:

• Quantize a 7B parameter model to 4-bit for deployment on consumer GPUs with limited VRAM.
• Reduce model size and memory footprint for faster downloads and lower storage costs.
• Achieve faster inference on A100 GPUs while maintaining acceptable perplexity compared to full-precision models.
• Fine-tune quantized models using peft adapters for domain-specific tasks without full model retraining.
• Export quantized models to Hugging Face Hub for team sharing and reproducible deployments.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Quantizes large language models to lower bit-widths using the GPTQ algorithm, reducing model size and memory requirements while enabling faster inference on consumer and enterprise GPUs.

Yes, if you need to quantize models now and can tolerate an archived codebase. AutoGPTQ is stable and widely used in production, with no known security vulnerabilities and permissive licensing. However, install with caution: the repository is archived (no updates since March 2024), so you will not receive compatibility fixes if torch, transformers, or CUDA versions change. Suitable for short-term projects or as a reference implementation; for long-term production use, verify that upstream dependencies remain compatible or plan to migrate to an actively maintained quantization framework.

Install

auto-gptq on PyPI

pip install auto-gptq

uv add auto-gptq

poetry add auto-gptq

Installing auto-gptq

Before you install

Medium friction: requires 11 runtime dependencies including torch, transformers, and accelerate. Pre-built wheels exist for Python 3.8–3.11 on Linux and Windows, but the package is archived and has not been updated since March 2024, raising maintenance concerns for future compatibility.

License in practice

Licensed under MIT (permissive), allowing commercial and private use with minimal restrictions, though the archived status means no ongoing legal or security updates from the maintainer.

Quickstart

pip install auto-gptq
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")
quantize_config = BaseQuantizeConfig(bits=4, group_size=128)
model = AutoGPTQForCausalLM.from_pretrained("facebook/opt-125m", quantize_config)
model.quantize([tokenizer("sample text")])
model.save_quantized("./quantized-model")

Requires CUDA 11.8, 12.1, or ROCm 5.7 and a compatible GPU; CPU-only installation is not supported. Building from source requires C++ compilation tools and may need ROCM_VERSION environment variable on AMD systems.

Verify before relying

• Whether the archived repository will receive security patches or compatibility fixes for future PyTorch/transformers releases.
• Compatibility with quantization formats or inference backends released after the final March 2024 update.
• Performance impact of using the fallback Python implementation when CUDA extensions are disabled.
• Actual inference speedup percentages on various GPU models and batch sizes beyond the documented token/s benchmarks.

Package facts

Evidence: auto_gptq-0.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; auto_gptq-0.7.1-cp310-cp310-win_amd64.whl; auto_gptq-0.7.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; auto_gptq-0.7.1-cp311-cp311-win_amd64.whl; auto_gptq-0.7.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; auto_gptq-0.7.1-cp38-cp38-win_amd64.whl; auto_gptq-0.7.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; auto_gptq-0.7.1-cp39-cp39-win_amd64.whl

Keywords: gptq, quantization, large-language-models, transformers

Tags

More Artificial Intelligence packages

Further reading