---
id: auto-round
version: "0.14.2"
license: Apache 2.0
license_treatment: permissive
maintenance: active
---

# auto-round — Repository of AutoRound: Advanced Weight-Only Quantization Algorithm for LLMs

License: permissive · Maintenance: active · Downloads: 300.1K/mo

## What it is and what it does

AutoRound is a quantization toolkit that compresses large language models and vision-language models to ultra-low bit widths (2–4 bits) while preserving accuracy. It uses sign-gradient descent to find optimal quantization parameters with minimal tuning overhead. The package integrates with popular inference frameworks (vLLM, SGLang, Transformers) and supports multiple export formats (AutoRound, AutoAWQ, AutoGPTQ, GGUF), making quantized models portable across different deployment environments.

The toolkit offers several quantization recipes—from fast round-to-nearest (RTN) baseline to more accurate iterative methods—and includes utilities for multi-GPU quantization, mixed-precision schemes, and multiple calibration datasets. It targets both researchers optimizing model accuracy at low bits and practitioners seeking to reduce model size and inference latency for deployment.

Use it for:

- Compress a 7B LLM to 2–3 bits for edge deployment or cost-effective cloud inference
- Export a quantized model to GGUF format for use with llama.cpp or other C++ inference engines
- Quantize a vision-language model for efficient multimodal inference on resource-constrained hardware
- Generate a mixed-precision quantization scheme automatically to balance accuracy and model size
- Integrate quantized models into vLLM or SGLang for fast batch inference with reduced memory footprint

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

AutoRound quantizes large language models and vision-language models to 2–4 bits with minimal accuracy loss, using sign-gradient descent and supporting multiple export formats and inference backends.

Yes. AutoRound is actively maintained, has no known vulnerabilities, installs with low friction, and is permissively licensed. It is well-suited for anyone needing to compress LLMs or VLMs for inference—whether for research, edge deployment, or cost reduction. Start with the CLI recipes (auto-round-best, auto-round-rtn) if you want quick results, or use the Python API for fine-grained control.

## Install

pip install auto-round
uv add auto-round
poetry add auto-round

## Installing auto-round

Before you install: Low friction install with a pure-Python wheel. The package is actively maintained (last commit 2026-08-14, 32 days since release) and depends on standard ML libraries (torch, transformers, datasets, numpy, accelerate, tqdm, py-cpuinfo, pydantic). Requires Python 3.10 or later.

License in practice: Apache 2.0 permissive license allows commercial and private use with minimal restrictions—suitable for most production and research contexts.

Quickstart:

pip install auto-round

from auto_round import AutoRound
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "meta-llama/Llama-2-7b"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

auto_round = AutoRound(model, tokenizer, dataset="wikitext2")
quantized_model = auto_round.quantize()

Requires Python 3.10+. Quantization is GPU-accelerated; CPU-only quantization is possible but slower. Needs torch and transformers installed.

Verify before relying:

- Exact memory overhead for mixed-precision scheme generation (stated as 1.1X–1.5X BF16 RAM but not verified independently)
- Actual quantization time for 7B models on single GPU (stated as ~10 minutes but hardware-dependent)
- Support matrix for all 10+ VLMs mentioned (list not provided in fact sheet)

## Package facts

- License: Apache 2.0 (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 300.1K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

LLM quantization low bit, model compression 2-4 bits, weight quantization toolkit, neural network quantization, efficient LLM inference, model size reduction, quantized model export, model-compression, quantization, llm-inference

[View on SkillFed](https://skillfed.io/packages/auto-round) · [View on PyPI](https://pypi.org/project/auto-round/)