nvidia-modelopt

What it is and what it does

NVIDIA Model Optimizer is a library for compressing and accelerating deep learning models through a suite of optimization techniques. It accepts PyTorch, Hugging Face, or ONNX models as input, applies techniques like quantization (FP8, NVFP4), pruning, distillation, and Neural Architecture Search to reduce model size and latency, then exports optimized checkpoints ready for deployment in inference frameworks like TensorRT-LLM, vLLM, and SGLang.

The package is designed for teams optimizing large language models and vision models for production inference. It integrates with NVIDIA Megatron-LM and Hugging Face Accelerate for training-time optimization, and provides Python APIs to compose multiple techniques together. Dependencies include torch, numpy, omegaconf, pydantic, and other ML-stack standards.

Use it for:

• Quantize large language models to lower precision (FP8, NVFP4) for faster inference without retraining.
• Prune and distill LLMs to reduce model size while maintaining accuracy for deployment on resource-constrained hardware.
• Apply post-training quantization to Hugging Face transformer models for immediate export to TensorRT or vLLM.
• Combine multiple optimization techniques (pruning + distillation + quantization) in a single workflow for maximum throughput gains.
• Export optimized diffusion models for faster image generation on NVIDIA GPUs.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Applies state-of-the-art model optimization techniques—quantization, pruning, Neural Architecture Search, distillation, speculative decoding, and sparsity—to accelerate PyTorch, Hugging Face, and ONNX models for deployment.

Yes, if you deploy PyTorch or Hugging Face models on NVIDIA hardware and need to reduce inference latency or memory footprint. The library is actively maintained, permissively licensed, and integrates cleanly with the NVIDIA inference ecosystem. Install friction is low and security record is clean. Not applicable for CPU-only or non-NVIDIA deployments.

Install

nvidia-modelopt on PyPI

pip install nvidia-modelopt

uv add nvidia-modelopt

poetry add nvidia-modelopt

Installing nvidia-modelopt

Before you install

Low friction install; pure Python wheel. Active maintenance with recent commits and 3442 repository stars. Depends on 15 runtime packages including torch, numpy, and omegaconf—standard for ML workflows.

License in practice

Apache-2.0 permissive license allows commercial use, modification, and distribution with minimal restrictions.

Quickstart

pip install nvidia-modelopt

from nvidia_modelopt import quantization
import torch

model = torch.nn.Linear(10, 5)
quantized = quantization.quantize(model)

Requires PyTorch and NVIDIA GPU support; quantization and optimization workflows typically need CUDA-capable hardware for practical speedup.

Verify before relying

• Whether the package requires NVIDIA-specific GPU hardware or works on CPU for development/testing.
• Performance gains and memory reduction percentages claimed in the description (e.g., 2.6× throughput) for typical model sizes.
• Compatibility with specific downstream frameworks (TensorRT-LLM, vLLM, SGLang) beyond the documented integration points.

Package facts

Evidence: nvidia_modelopt-0.45.0-py3-none-any.whl

Tags

More Artificial Intelligence packages

Further reading