auto-round

What it is and what it does

AutoRound is a quantization toolkit that compresses large language models and vision-language models to ultra-low bit widths (2–4 bits) while preserving accuracy. It uses sign-gradient descent to find optimal quantization parameters with minimal tuning overhead. The package integrates with popular inference frameworks (vLLM, SGLang, Transformers) and supports multiple export formats (AutoRound, AutoAWQ, AutoGPTQ, GGUF), making quantized models portable across different deployment environments.

The toolkit offers several quantization recipes—from fast round-to-nearest (RTN) baseline to more accurate iterative methods—and includes utilities for multi-GPU quantization, mixed-precision schemes, and multiple calibration datasets. It targets both researchers optimizing model accuracy at low bits and practitioners seeking to reduce model size and inference latency for deployment.

Use it for:

• Compress a 7B LLM to 2–3 bits for edge deployment or cost-effective cloud inference
• Export a quantized model to GGUF format for use with llama.cpp or other C++ inference engines
• Quantize a vision-language model for efficient multimodal inference on resource-constrained hardware
• Generate a mixed-precision quantization scheme automatically to balance accuracy and model size
• Integrate quantized models into vLLM or SGLang for fast batch inference with reduced memory footprint

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

AutoRound quantizes large language models and vision-language models to 2–4 bits with minimal accuracy loss, using sign-gradient descent and supporting multiple export formats and inference backends.

Yes. AutoRound is actively maintained, has no known vulnerabilities, installs with low friction, and is permissively licensed. It is well-suited for anyone needing to compress LLMs or VLMs for inference—whether for research, edge deployment, or cost reduction. Start with the CLI recipes (auto-round-best, auto-round-rtn) if you want quick results, or use the Python API for fine-grained control.

Install

auto-round on PyPI

pip install auto-round

uv add auto-round

poetry add auto-round

Installing auto-round

Before you install

Low friction install with a pure-Python wheel. The package is actively maintained (last commit 2026-08-14, 32 days since release) and depends on standard ML libraries (torch, transformers, datasets, numpy, accelerate, tqdm, py-cpuinfo, pydantic). Requires Python 3.10 or later.

License in practice

Apache 2.0 permissive license allows commercial and private use with minimal restrictions—suitable for most production and research contexts.

Quickstart

pip install auto-round

from auto_round import AutoRound
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "meta-llama/Llama-2-7b"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

auto_round = AutoRound(model, tokenizer, dataset="wikitext2")
quantized_model = auto_round.quantize()

Requires Python 3.10+. Quantization is GPU-accelerated; CPU-only quantization is possible but slower. Needs torch and transformers installed.

Verify before relying

• Exact memory overhead for mixed-precision scheme generation (stated as 1.1X–1.5X BF16 RAM but not verified independently)
• Actual quantization time for 7B models on single GPU (stated as ~10 minutes but hardware-dependent)
• Support matrix for all 10+ VLMs mentioned (list not provided in fact sheet)

Package facts

Evidence: auto_round-0.14.2-py3-none-any.whl

Keywords: quantization, auto-around, LLM, SignRound

Tags

More Artificial Intelligence packages

Further reading