---
id: torchao
version: "0.18.0"
license: unclear
license_treatment: unclear
maintenance: active
---

# torchao — Package for applying ao techniques to GPU models

License: unclear · Maintenance: active · Downloads: 3.7M/mo

## What it is and what it does

TorchAO is a quantization and sparsity library that reduces model size and accelerates both training and inference by converting weights and activations to lower-precision formats (int4, float8, etc.) and applying structured sparsity patterns. It integrates directly into compilation and distributed training pipelines, allowing you to apply these optimizations to large language models and other neural networks with minimal code changes.

The library supports multiple quantization strategies—weight-only quantization for inference, dynamic quantization for both training and serving, and quantization-aware training (QAT) to recover accuracy lost during quantization. It is designed to work out-of-the-box with HuggingFace Transformers models and has been validated on large-scale training runs, achieving reported speedups with acceptable accuracy trade-offs.

Use it for:

- Reduce inference latency and memory footprint of large language models by quantizing weights to int4 or float8 for deployment on resource-constrained hardware.
- Accelerate multi-GPU pre-training of large models by applying float8 training with FSDP2 to reduce communication overhead and memory pressure.
- Fine-tune quantized models using quantization-aware training (QAT) to recover accuracy degradation from aggressive quantization.
- Apply semi-structured 2:4 sparsity patterns to transformer models to achieve training and inference speedups with minimal accuracy loss.
- Integrate quantization into existing HuggingFace Transformers workflows via the TorchAoConfig API without rewriting model loading code.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

TorchAO applies quantization and sparsity techniques to PyTorch models for faster training and inference with reduced memory usage, working natively with torch.compile() and FSDP2.

Yes, if you are optimizing models for inference speed or training efficiency and can tolerate the license ambiguity. The package is actively maintained, has no external runtime dependencies, and is widely adopted in production serving. Verify the license terms in the repository before use in proprietary contexts; otherwise, install friction is minimal and performance gains are substantial.

## Install

pip install torchao
uv add torchao
poetry add torchao

## Installing torchao

Before you install: Low install friction; ships as a pure Python wheel. Active maintenance with a recent release (11 days old) and steady commit history. No runtime dependencies to manage.

License in practice: License status is unclear from the package metadata—no SPDX identifier or raw license text is recorded. Verify the actual license terms in the repository before adopting in proprietary or restricted contexts.

Quickstart:

pip install torchao

from torchao.quantization import Int4WeightOnlyConfig, quantize_

quantize_(model, Int4WeightOnlyConfig(group_size=32))

Requires a compatible PyTorch installation and GPU or CPU setup; optional accelerated kernels improve performance for specific workflows but are not required for basic quantization.

Verify before relying:

- Exact Python version compatibility and minimum PyTorch version requirements
- Whether the package is suitable for production deployment or remains research-focused
- Specific hardware platforms beyond NVIDIA GPUs where quantization techniques are validated

## Package facts

- License: not declared (unclear)
- Python support: unspecified
- Install friction: low
- Maintenance: active
- Downloads: 3.7M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

pytorch model quantization, int4 weight quantization, float8 training optimization, llm inference speedup, neural network compression, model sparsity pytorch, quantization aware training, model-optimization, quantization, llm-inference

[View on SkillFed](https://skillfed.io/packages/torchao) · [View on PyPI](https://pypi.org/project/torchao/)