--- id: genai-perf version: "0.0.16" license: BSD license_treatment: permissive maintenance: active --- # genai-perf — GenAI Perf Analyzer CLI - CLI tool to simplify profiling LLMs and Generative AI models with Perf Analyzer License: permissive · Maintenance: active · Downloads: 314.5K/mo ## What it is and what it does GenAI-Perf is a profiling and benchmarking tool designed to measure the performance characteristics of generative AI models running on inference servers. It generates configurable load (concurrent requests or request rates) against a running inference server and collects detailed metrics including output token throughput, time to first token, inter-token latency, and request latency. Results are reported in console tables and exported to CSV and JSON for further analysis. The tool targets a wide range of model types—large language models, multi-modal models, embeddings, ranking models, and LoRA-adapted variants—and supports both synthetic load generation and real input datasets. It can be configured via command-line arguments or YAML configuration files, and provides customizable frontends and Jinja2-templated payloads for benchmarking custom APIs. The package is in active development (Alpha status) and requires an external inference server to already be running. Use it for: - Benchmark LLM inference latency and throughput on Triton Inference Server with TensorRT-LLM backends. - Measure time-to-first-token and inter-token latency for streaming language model deployments. - Profile multi-modal model performance under concurrent request loads to identify bottlenecks. - Compare inference performance across different model backends or hardware configurations. - Generate performance reports (CSV/JSON) for embedding or ranking models to track optimization progress. - Test custom API endpoints with templated payloads to validate inference server integration. ## Worth the install? AI-flagged interpretation of the facts above — verify before relying. GenAI-Perf is a command-line tool for measuring throughput, latency, and token-generation metrics of generative AI models served through an inference server, supporting LLMs, multi-modal models, embeddings, and custom APIs. Yes, if you are benchmarking generative AI models on Triton Inference Server or compatible inference servers and need detailed token-level and request-level metrics. The low install friction and active maintenance make it a practical choice. Requires CUDA 12 and a running inference server; the large dependency tree (19 runtime packages) may add setup time. Not suitable if you need to profile models without an external inference server or on non-Triton platforms. ## Install pip install genai-perf uv add genai-perf poetry add genai-perf ## Installing genai-perf Before you install: Low install friction with a pure Python wheel. Requires Python 3.10 or 3.12 and CUDA 12 to be pre-installed; the 19 runtime dependencies include heavy data science and ML stacks (transformers, pandas, numpy, plotly, statsmodels) which may take time to resolve. Actively maintained with recent commits. License in practice: BSD permissive license allows commercial and private use with minimal restrictions; you must retain copyright notices and disclaimers in redistributions. Quickstart: pip install genai-perf genai-perf profile -m gpt2 --backend tensorrtllm --streaming Requires CUDA 12 to be pre-installed on the system and an inference server (e.g., Triton) already running at the specified endpoint. Verify before relying: - Whether the tool works with inference servers other than Triton Inference Server. - Performance overhead of the profiling tool itself on measured metrics. - Compatibility with custom model backends beyond those documented in the description. ## Package facts - License: BSD (permissive) - Python support: supports_current - Install friction: low - Maintenance: active - Downloads: 314.5K/month (top 15,000 on PyPI) - Known vulnerabilities: none known ## Tags llm performance benchmarking, inference server latency measurement, token throughput profiling, generative ai model metrics, perf analyzer cli tool, time to first token measurement, request latency benchmarking, benchmarking, inference-server, llm-profiling [View on SkillFed](https://skillfed.io/packages/genai-perf) · [View on PyPI](https://pypi.org/project/genai-perf/)