auto-gptq
An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.
Decision gist · record as of 2026-08-14
Yes, if you need to quantize models now and can tolerate an archived codebase. AutoGPTQ is stable and widely used in production, with no known security vulnerabilities and permissive licensing. However, install with caution: the repository is archived (no updates since March 2024), so you will not receive compatibility fixes if torch, transformers, or CUDA versions change. Suitable for short-term projects or as a reference implementation; for long-term production use, verify that upstream dependencies remain compatible or plan to migrate to an actively maintained quantization framework.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires CUDA 11.8, 12.1, or ROCm 5.7 and a compatible GPU; CPU-only installation is not supported.
- Building from source requires C++ compilation tools and may need ROCM_VERSION environment variable on AMD systems.
- Medium friction: requires 11 runtime dependencies including torch, transformers, and accelerate.
License · maintenance · safety
permissive license (permissive) — Licensed under MIT (permissive), allowing commercial and private use with minimal restrictions, though the archived status means no ongoing legal or security updates from the maintainer.
last release 2024-03-01 (896 days) · last repo commit 2025-04-11 · 5,071 stars · archived
0 known vulnerabilities (OSV.dev, 2026-08-14) · 99,069 downloads/mo, #13,041 on PyPI
Alternatives
Verify before relying
pip install auto-gptq
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")
quantize_config = BaseQuantizeConfig(bits=4, group_size=128)
model = AutoGPTQForCausalLM.from_pretrained("facebook/opt-125m", quantize_config)
model.quantize([tokenizer("sample text")])
model.save_quantized("./quantized-model")- Whether the archived repository will receive security patches or compatibility fixes for future PyTorch/transformers releases.
- Compatibility with quantization formats or inference backends released after the final March 2024 update.
- Performance impact of using the fallback Python implementation when CUDA extensions are disabled.
- Actual inference speedup percentages on various GPU models and batch sizes beyond the documented token/s benchmarks.
What it is and what it does
AutoGPTQ is a quantization library that compresses large language models to 4-bit or lower precision using the GPTQ weight-only quantization algorithm. It integrates with transformers and provides APIs to quantize models, save them in compressed form, and load them for inference on GPUs. The library is designed to reduce memory footprint and accelerate inference speed—the documentation shows examples where quantized models achieve 25.53 tokens/second on A100 GPUs compared to 18.87 tokens/second for full-precision equivalents.
The package depends on a substantial stack: torch, transformers, peft, accelerate, datasets, and several utility libraries. It supports CUDA 11.8, 12.1, and ROCm 5.7 via pre-built wheels for Python 3.8–3.11 on Linux and Windows. However, the repository is now archived and has not been updated since March 2024, meaning no new features, bug fixes, or compatibility patches are expected. Users relying on this package should be prepared to maintain it themselves or migrate to actively maintained alternatives if breaking changes occur in upstream dependencies.
Use it for
- Quantize a 7B parameter model to 4-bit for deployment on consumer GPUs with limited VRAM.
- Reduce model size and memory footprint for faster downloads and lower storage costs.
- Achieve faster inference on A100 GPUs while maintaining acceptable perplexity compared to full-precision models.
- Fine-tune quantized models using peft adapters for domain-specific tasks without full model retraining.
- Export quantized models to Hugging Face Hub for team sharing and reproducible deployments.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you need to quantize models now and can tolerate an archived codebase.
AutoGPTQ is stable and widely used in production, with no known security vulnerabilities and permissive licensing. However, install with caution: the repository is archived (no updates since March 2024), so you will not receive compatibility fixes if torch, transformers, or CUDA versions change. Suitable for short-term projects or as a reference implementation; for long-term production use, verify that upstream dependencies remain compatible or plan to migrate to an actively maintained quantization framework.
Install
auto-gptq on PyPI
Before you install
Medium friction: requires 11 runtime dependencies including torch, transformers, and accelerate. Pre-built wheels exist for Python 3.8–3.11 on Linux and Windows, but the package is archived and has not been updated since March 2024, raising maintenance concerns for future compatibility.
Requires CUDA 11.8, 12.1, or ROCm 5.7 and a compatible GPU; CPU-only installation is not supported. Building from source requires C++ compilation tools and may need ROCM_VERSION environment variable on AMD systems.
License in practice
Licensed under MIT (permissive), allowing commercial and private use with minimal restrictions, though the archived status means no ongoing legal or security updates from the maintainer.
Quickstart
pip install auto-gptq
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")
quantize_config = BaseQuantizeConfig(bits=4, group_size=128)
model = AutoGPTQForCausalLM.from_pretrained("facebook/opt-125m", quantize_config)
model.quantize([tokenizer("sample text")])
model.save_quantized("./quantized-model")Verify before relying
- Whether the archived repository will receive security patches or compatibility fixes for future PyTorch/transformers releases.
- Compatibility with quantization formats or inference backends released after the final March 2024 update.
- Performance impact of using the fallback Python implementation when CUDA extensions are disabled.
- Actual inference speedup percentages on various GPU models and batch sizes beyond the documented token/s benchmarks.
Package facts
| License | permissive license permissive |
| Python support | Supports the current Python release >=3.8.0 |
| Install friction | Medium. Platform-specific wheel |
| Runtime dependencies | 11 packagesacceleratedatasetssentencepiecenumpyrougegekkotorchsafetensorstransformerspefttqdm |
| Maintenance | Abandoned 896 days since the last release |
| Last repo commit | repository archived |
| First released | |
| Downloads | 99,069 / month, #13,041 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Environment :: GPU :: NVIDIA CUDA :: 11.7Environment :: GPU :: NVIDIA CUDA :: 11.8Environment :: GPU :: NVIDIA CUDA :: 12License :: OSI Approved :: MIT LicenseNatural Language :: Chinese (Simplified)Natural Language :: EnglishProgramming Language :: C++Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.8Programming Language :: Python :: 3.9 |
Evidence: auto_gptq-0.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; auto_gptq-0.7.1-cp310-cp310-win_amd64.whl; auto_gptq-0.7.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; auto_gptq-0.7.1-cp311-cp311-win_amd64.whl; auto_gptq-0.7.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; auto_gptq-0.7.1-cp38-cp38-win_amd64.whl; auto_gptq-0.7.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; auto_gptq-0.7.1-cp39-cp39-win_amd64.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “llm quantization gptq”
- auto-gptqQuantizes large language models to lower bit-widths using the GPTQ…
- compressed-tensorsProvides a unified format for storing and loading compressed neural…
- ms-swiftms-swift is a framework for training, evaluating, quantizing, and…
Give your agent the search over MCP, or paste the wish link into any chat.
More Artificial Intelligence packages
LiteLLM provides a unified Python interface to call 100+ LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Azure, and others) using OpenAI-compatible API format, available as both a Python SDK and a self-hosted AI Gateway proxy server.
Install it if you need to work with multiple LLM providers or want to centralize LLM routing in your organization.
Client library and CLI tool for downloading, uploading, and managing models, datasets, and repositories on the Hugging Face Hub platform.
Install it if you work with Hugging Face Hub models or datasets.
LangChain provides a framework for building agents and LLM-powered applications by composing language models, tools, and memory through a unified API that abstracts over multiple model providers.
hf-xet provides chunk-based deduplication and efficient file transfer for the Hugging Face Hub, enabling faster uploads and downloads of large files with local disk caching.
Tokenizers converts raw text into token sequences for NLP models, with support for training custom vocabularies and using pre-built tokenizers (BPE, WordPiece) optimized for speed via Rust.
Transformers provides a unified framework for loading, fine-tuning, and running state-of-the-art pretrained models across text, vision, audio, video, and multimodal tasks using PyTorch, JAX, or TensorFlow.
Install it if you need to run or train any transformer-based model for NLP, vision, audio, or multimodal tasks.
See also llmcompressor · humming-kernels · nvidia-modelopt · auto-round · ai-edge-quantizer · model-compression-toolkit · torchao · diffq · mmgp · ctranslate2