aiperf

What it is and what it does

AIPerf is a benchmarking tool that loads generative AI inference endpoints with configurable request patterns and collects detailed performance metrics. It measures time-to-first-token, inter-token latency, request latency, token throughput, and sequence lengths, presenting results in real-time dashboards or static reports. The tool uses a multiprocess architecture with 10 services communicating via ZMQ, supporting multiple UI modes (dashboard, simple progress bars, or headless) and benchmarking strategies including concurrency ramping, request-rate control, trace replay, and adaptive scaling.

You provide an endpoint URL, model name, and workload parameters (concurrency, request count, or arrival patterns), and AIPerf drives load against it while collecting metrics. It integrates with public datasets like ShareGPT, supports OpenAI-compatible chat and completion APIs as well as NIM embeddings and rankings, and offers optional telemetry streaming to MLflow, OpenTelemetry, or Weights & Biases. Results export to CSV, JSON, and logs for further analysis.

Use it for:

• Profile latency and throughput of a vLLM or TGI deployment under varying concurrency to find performance bottlenecks
• Replay production traffic traces (Bailian, Baseten, BurstGPT, SageMaker) to benchmark how an endpoint handles real-world workloads
• Discover SLA boundaries by running adaptive scaling to find the maximum sustainable concurrency or request rate
• Compare token-level metrics (TTFT, ITL, TPS) across different model sizes or inference frameworks on the same hardware
• Test request cancellation and timeout resilience by configuring timeout thresholds and observing endpoint behavior

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

AIPerf measures the performance of generative AI models served by inference endpoints, collecting detailed latency, throughput, and token metrics via command-line benchmarking and generating comprehensive reports.

Yes. AIPerf is actively maintained, has no known vulnerabilities, and provides a comprehensive benchmarking solution for generative AI inference endpoints. Install friction is low on common platforms (wheels available for x86_64 and macOS); Linux aarch64 users need a C compiler. The permissive Apache-2.0 license poses no barrier. Install it if you need to measure or compare LLM inference performance across endpoints, models, or load patterns.

Install

aiperf on PyPI

pip install aiperf

uv add aiperf

poetry add aiperf

Installing aiperf

Before you install

Low install friction with pre-built wheels on Linux x86_64, macOS, and Windows. On Linux aarch64, the crick dependency requires a C compiler and build toolchain at install time. Package is actively maintained with a recent release.

License in practice

Apache-2.0 permissive license allows use in commercial and proprietary projects with minimal restrictions; you must include a copy of the license and note any modifications.

Quickstart

pip install aiperf

aiperf profile \
  --model "granite4:350m" \
  --streaming \
  --endpoint-type chat \
  --tokenizer ibm-granite/granite-4.0-micro \
  --url http://localhost:11434 \
  --request-count 10

On Linux aarch64, install build-essential (Debian/Ubuntu) or Development Tools (RHEL/CentOS) before pip install aiperf, as the crick dependency requires a C compiler.

Verify before relying

• Whether optional telemetry integrations (MLflow, OpenTelemetry, Weights & Biases) are production-ready or experimental
• Performance overhead of the multiprocess ZMQ architecture on small benchmarks
• Compatibility with inference endpoints beyond OpenAI and NIM APIs

Package facts

Evidence: aiperf-0.12.0-py3-none-any.whl

Tags

More Testing packages

Further reading