genai-perf

What it is and what it does

GenAI-Perf is a profiling and benchmarking tool designed to measure the performance characteristics of generative AI models running on inference servers. It generates configurable load (concurrent requests or request rates) against a running inference server and collects detailed metrics including output token throughput, time to first token, inter-token latency, and request latency. Results are reported in console tables and exported to CSV and JSON for further analysis.

The tool targets a wide range of model types—large language models, multi-modal models, embeddings, ranking models, and LoRA-adapted variants—and supports both synthetic load generation and real input datasets. It can be configured via command-line arguments or YAML configuration files, and provides customizable frontends and Jinja2-templated payloads for benchmarking custom APIs. The package is in active development (Alpha status) and requires an external inference server to already be running.

Use it for:

• Benchmark LLM inference latency and throughput on Triton Inference Server with TensorRT-LLM backends.
• Measure time-to-first-token and inter-token latency for streaming language model deployments.
• Profile multi-modal model performance under concurrent request loads to identify bottlenecks.
• Compare inference performance across different model backends or hardware configurations.
• Generate performance reports (CSV/JSON) for embedding or ranking models to track optimization progress.
• Test custom API endpoints with templated payloads to validate inference server integration.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

GenAI-Perf is a command-line tool for measuring throughput, latency, and token-generation metrics of generative AI models served through an inference server, supporting LLMs, multi-modal models, embeddings, and custom APIs.

Yes, if you are benchmarking generative AI models on Triton Inference Server or compatible inference servers and need detailed token-level and request-level metrics. The low install friction and active maintenance make it a practical choice. Requires CUDA 12 and a running inference server; the large dependency tree (19 runtime packages) may add setup time. Not suitable if you need to profile models without an external inference server or on non-Triton platforms.

Install

genai-perf on PyPI

pip install genai-perf

uv add genai-perf

poetry add genai-perf

Installing genai-perf

Before you install

Low install friction with a pure Python wheel. Requires Python 3.10 or 3.12 and CUDA 12 to be pre-installed; the 19 runtime dependencies include heavy data science and ML stacks (transformers, pandas, numpy, plotly, statsmodels) which may take time to resolve. Actively maintained with recent commits.

License in practice

BSD permissive license allows commercial and private use with minimal restrictions; you must retain copyright notices and disclaimers in redistributions.

Quickstart

pip install genai-perf
genai-perf profile -m gpt2 --backend tensorrtllm --streaming

Requires CUDA 12 to be pre-installed on the system and an inference server (e.g., Triton) already running at the specified endpoint.

Verify before relying

• Whether the tool works with inference servers other than Triton Inference Server.
• Performance overhead of the profiling tool itself on measured metrics.
• Compatibility with custom model backends beyond those documented in the description.

Package facts

Evidence: genai_perf-0.0.16-py3-none-any.whl

Tags

More Software Development packages

Further reading