sit4onnx

What it is and what it does

sit4onnx is a command-line and Python API tool for testing ONNX model inference across different hardware accelerators and execution providers. It accepts an ONNX model file and runs it repeatedly to measure average inference latency, with options to specify batch sizes, fixed input shapes, and custom test data via numpy arrays or .npy files. The tool supports TensorRT, CUDA, OpenVINO (CPU and GPU), and CPU-only execution, and can output performance profiling results and inference outputs to files.

The package is designed for model validation and benchmarking workflows where you need to verify that a model runs correctly on a target platform and measure its throughput or latency. It handles models with dynamic input shapes by allowing you to specify fixed dimensions, and it returns the final inference result as a numpy array when called from Python code.

Use it for:

• Benchmark an ONNX model's inference speed on TensorRT or CUDA before deploying to production.
• Validate that a model with dynamic input shapes runs correctly by specifying fixed dimensions and test data.
• Compare inference latency across different execution providers (CPU vs. GPU vs. TensorRT) on the same model.
• Profile a model's performance and export timing results to JSON for analysis and reporting.
• Test a model end-to-end with custom numpy array inputs to verify output correctness before integration.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Runs inference tests on ONNX models using TensorRT, CUDA, OpenVINO, or CPU execution providers, measuring performance and validating model outputs.

Yes, if you need to test and benchmark ONNX models across different execution providers. The low install friction and permissive license make it a practical choice. However, note that the package is aging (last commit 341 days ago) and you will need to separately install the execution provider libraries (TensorRT, OpenVINO, etc.) that you intend to use.

Install

sit4onnx on PyPI

pip install sit4onnx

uv add sit4onnx

poetry add sit4onnx

Installing sit4onnx

Before you install

Low install friction with no runtime dependencies. Maintenance shows aging status—last commit was 341 days ago—but the repository remains active and unarchived.

License in practice

MIT License permits commercial and private use with minimal restrictions, making it suitable for most projects.

Quickstart

pip install sit4onnx

from sit4onnx import inference

results = inference(
    input_onnx_file_path="model.onnx",
    batch_size=10,
    test_loop_count=10,
    onnx_execution_provider="tensorrt"
)

Requires the target execution provider library (TensorRT, CUDA, OpenVINO, or onnxruntime CPU) to be installed separately; sit4onnx does not bundle these dependencies.

Verify before relying

• Whether onnxruntime or execution provider libraries (TensorRT, OpenVINO) are automatically installed or must be provided separately.
• Supported versions of TensorRT, CUDA, and OpenVINO beyond the requirement that Python be >=3.6.
• Whether profiling output format and structure are documented beyond the mention of .json files.

Package facts

Evidence: sit4onnx-1.0.10-py3-none-any.whl

Tags

More Testing packages