torchao

What it is and what it does

TorchAO is a quantization and sparsity library that reduces model size and accelerates both training and inference by converting weights and activations to lower-precision formats (int4, float8, etc.) and applying structured sparsity patterns. It integrates directly into compilation and distributed training pipelines, allowing you to apply these optimizations to large language models and other neural networks with minimal code changes.

The library supports multiple quantization strategies—weight-only quantization for inference, dynamic quantization for both training and serving, and quantization-aware training (QAT) to recover accuracy lost during quantization. It is designed to work out-of-the-box with HuggingFace Transformers models and has been validated on large-scale training runs, achieving reported speedups with acceptable accuracy trade-offs.

Use it for:

• Reduce inference latency and memory footprint of large language models by quantizing weights to int4 or float8 for deployment on resource-constrained hardware.
• Accelerate multi-GPU pre-training of large models by applying float8 training with FSDP2 to reduce communication overhead and memory pressure.
• Fine-tune quantized models using quantization-aware training (QAT) to recover accuracy degradation from aggressive quantization.
• Apply semi-structured 2:4 sparsity patterns to transformer models to achieve training and inference speedups with minimal accuracy loss.
• Integrate quantization into existing HuggingFace Transformers workflows via the TorchAoConfig API without rewriting model loading code.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

TorchAO applies quantization and sparsity techniques to PyTorch models for faster training and inference with reduced memory usage, working natively with torch.compile() and FSDP2.

Yes, if you are optimizing models for inference speed or training efficiency and can tolerate the license ambiguity. The package is actively maintained, has no external runtime dependencies, and is widely adopted in production serving. Verify the license terms in the repository before use in proprietary contexts; otherwise, install friction is minimal and performance gains are substantial.

Install

torchao on PyPI

pip install torchao

uv add torchao

poetry add torchao

Installing torchao

Before you install

Low install friction; ships as a pure Python wheel. Active maintenance with a recent release (11 days old) and steady commit history. No runtime dependencies to manage.

License in practice

License status is unclear from the package metadata—no SPDX identifier or raw license text is recorded. Verify the actual license terms in the repository before adopting in proprietary or restricted contexts.

Quickstart

pip install torchao

from torchao.quantization import Int4WeightOnlyConfig, quantize_

quantize_(model, Int4WeightOnlyConfig(group_size=32))

Requires a compatible PyTorch installation and GPU or CPU setup; optional accelerated kernels improve performance for specific workflows but are not required for basic quantization.

Verify before relying

• Exact Python version compatibility and minimum PyTorch version requirements
• Whether the package is suitable for production deployment or remains research-focused
• Specific hardware platforms beyond NVIDIA GPUs where quantization techniques are validated

Package facts

Evidence: torchao-0.18.0-py3-none-any.whl

Tags

More Artificial Intelligence packages