onnxruntime-gpu

What it is and what it does

ONNX Runtime GPU is a performance-focused inference engine that executes Open Neural Network Exchange (ONNX) models on NVIDIA GPUs. It takes pre-trained models in ONNX format and runs them efficiently on GPU hardware, accelerating predictions for machine learning workloads. The package depends on flatbuffers, numpy, packaging, and protobuf for model serialization, numerical operations, and dependency management.

This is the GPU-specific variant of ONNX Runtime, designed for scenarios where inference speed matters and GPU hardware is available. It's commonly used in production systems, real-time inference pipelines, and research environments where model throughput or latency is critical. The package is maintained by Microsoft, actively developed, and supports recent Python versions (3.11–3.14) across Linux and Windows platforms.

Use it for:

• Accelerate inference for computer vision models (object detection, image classification) in production systems.
• Run real-time NLP model inference (transformers, language models) with GPU speedup for low-latency predictions.
• Deploy quantized or optimized ONNX models on GPU for high-throughput batch inference in data pipelines.
• Execute recommendation system models on GPU for fast scoring in online serving environments.
• Benchmark or evaluate model performance on GPU hardware during model development and optimization.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Executes ONNX machine learning models on GPU hardware, providing inference acceleration for neural networks and other machine learning workloads.

Yes, if you have GPU hardware available and need to run ONNX models in production or research. The package is actively maintained, permissively licensed, and widely used (top 5000 PyPI). Install friction is moderate due to GPU driver/CUDA setup requirements, but the package itself is stable and well-supported. Not suitable if you lack GPU hardware or need CPU-only inference.

Install

onnxruntime-gpu on PyPI

pip install onnxruntime-gpu

uv add onnxruntime-gpu

poetry add onnxruntime-gpu

Installing onnxruntime-gpu

Before you install

Medium install friction due to GPU-specific wheels and binary dependencies. The package is actively maintained with a recent release (20 days old) and strong repository signals (21377 stars, active development). Supports Python 3.11 through 3.14 with platform-specific wheels for Linux and Windows.

License in practice

MIT License permits commercial and private use with minimal restrictions. You may use, modify, and distribute this package freely provided you include the license notice.

Quickstart

pip install onnxruntime-gpu

import onnxruntime as rt
sess = rt.InferenceSession('model.onnx')
results = sess.run(None, {'input_name': input_data})

Requires NVIDIA GPU with compatible CUDA/cuDNN libraries installed on the system; GPU drivers and CUDA toolkit must be configured separately.

Verify before relying

• Specific CUDA version requirements and GPU compatibility matrix not provided in fact sheet.
• Performance benchmarks or speedup claims relative to CPU inference not available.
• Whether quantization, optimization, or model conversion tools are included in this package.

Package facts

Evidence: onnxruntime_gpu-1.28.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl; onnxruntime_gpu-1.28.0-cp311-cp311-win_amd64.whl; onnxruntime_gpu-1.28.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl; onnxruntime_gpu-1.28.0-cp312-cp312-win_amd64.whl; onnxruntime_gpu-1.28.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl; onnxruntime_gpu-1.28.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl; onnxruntime_gpu-1.28.0-cp313-cp313-win_amd64.whl; onnxruntime_gpu-1.28.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl; onnxruntime_gpu-1.28.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl; onnxruntime_gpu-1.28.0-cp314-cp314-win_amd64.whl

Keywords: onnx, machine, learning

Tags

More Software Development packages

Further reading