onnxruntime-genai

What it is and what it does

ONNX Runtime GenAI is a Python library that wraps ONNX Runtime to simplify inference of language models and multi-modal models. It abstracts away the complexity of loading ONNX model files and managing inference sessions, letting you focus on running models rather than configuring the runtime. The library depends on numpy and onnxruntime, and is distributed as pre-built wheels across Windows, Linux, and macOS on both x86 and ARM architectures.

The package is intended for scenarios where you want to deploy or experiment with LLMs and SLMs locally or in cloud environments without relying on external APIs. It bridges the gap between raw ONNX Runtime (a low-level inference engine) and high-level model frameworks, offering a middle ground for developers who have ONNX-converted models and want a straightforward Python interface to run them.

Use it for:

• Deploy a quantized LLM locally on a laptop or edge device for inference without cloud API calls
• Run multi-modal models in a Python application using pre-converted ONNX weights
• Integrate on-device SLM inference into a web service or backend for low-latency responses
• Experiment with different model architectures by swapping ONNX model files without code changes
• Build a chatbot or text-generation pipeline that runs entirely on-premises for privacy

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Runs small and large language models and multi-modal models on-device and in the cloud using ONNX Runtime as the execution backend.

Yes, if you have ONNX-converted models and want a straightforward Python interface to run them on-device or in the cloud. The library is actively maintained, has no known vulnerabilities, and offers broad platform coverage. Install only if you already have or plan to convert models to ONNX format; it is not a model hub or download tool.

Install

onnxruntime-genai on PyPI

pip install onnxruntime-genai

uv add onnxruntime-genai

poetry add onnxruntime-genai

Installing onnxruntime-genai

Before you install

Medium install friction due to pre-built wheels for multiple Python versions and platforms. Active maintenance with a release 8 days old.

License in practice

MIT License permits commercial and private use with minimal restrictions; suitable for most projects.

Quickstart

pip install onnxruntime-genai
import onnxruntime_genai
# Load and run a model via the ONNX Runtime GenAI API

Requires numpy and onnxruntime as runtime dependencies; model files and platform-specific ONNX Runtime support are prerequisites.

Verify before relying

• Specific model formats and sizes supported by this version
• Performance characteristics (latency, throughput) on different hardware
• Whether quantization or optimization tooling is included
• API stability and backward compatibility guarantees
• Python version requirements (requires_python field is unspecified)

Package facts

Evidence: onnxruntime_genai-0.15.2-cp311-cp311-macosx_12_0_arm64.whl; onnxruntime_genai-0.15.2-cp311-cp311-manylinux_2_28_aarch64.whl; onnxruntime_genai-0.15.2-cp311-cp311-manylinux_2_28_x86_64.whl; onnxruntime_genai-0.15.2-cp311-cp311-win_amd64.whl; onnxruntime_genai-0.15.2-cp311-cp311-win_arm64.whl; onnxruntime_genai-0.15.2-cp312-cp312-macosx_12_0_arm64.whl; onnxruntime_genai-0.15.2-cp312-cp312-manylinux_2_28_aarch64.whl; onnxruntime_genai-0.15.2-cp312-cp312-manylinux_2_28_x86_64.whl; onnxruntime_genai-0.15.2-cp312-cp312-win_amd64.whl; onnxruntime_genai-0.15.2-cp312-cp312-win_arm64.whl; onnxruntime_genai-0.15.2-cp313-cp313-macosx_12_0_arm64.whl; onnxruntime_genai-0.15.2-cp313-cp313-manylinux_2_28_aarch64.whl; onnxruntime_genai-0.15.2-cp313-cp313-manylinux_2_28_x86_64.whl; onnxruntime_genai-0.15.2-cp313-cp313-win_amd64.whl; onnxruntime_genai-0.15.2-cp313-cp313-win_arm64.whl; onnxruntime_genai-0.15.2-cp314-cp314-macosx_12_0_arm64.whl; onnxruntime_genai-0.15.2-cp314-cp314-manylinux_2_28_aarch64.whl; onnxruntime_genai-0.15.2-cp314-cp314-manylinux_2_28_x86_64.whl; onnxruntime_genai-0.15.2-cp314-cp314-win_amd64.whl; onnxruntime_genai-0.15.2-cp314-cp314-win_arm64.whl

Tags

More Artificial Intelligence packages