---
id: nvidia-lm-eval
version: "26.3"
license: MIT
license_treatment: permissive
maintenance: active
---

# nvidia-lm-eval — A framework for evaluating language models - packaged by NVIDIA

License: permissive · Maintenance: active · Downloads: 127.3K/mo

## What it is and what it does

NVIDIA NeMo Evaluator is a framework for running standardized benchmarks against language model endpoints. It wraps the lm-evaluation-harness with NVIDIA's tooling to simplify evaluation workflows, providing a command-line interface and YAML-based configuration system. You point it at a model API (chat or completions style), specify a benchmark task (MMLU, IFEval, MATH, etc.), and it handles request batching, result collection, and output formatting.

The package is designed for teams evaluating models deployed as API endpoints rather than local inference. It depends on a large ecosystem of evaluation libraries (evaluate, datasets, rouge-score, sacrebleu, transformers) and integrates with OpenAI-compatible APIs. Configuration is hierarchical—command-line flags override YAML config, which overrides task defaults. Results are saved to a specified directory as YAML.

Use it for:

- Benchmark a custom or third-party LLM endpoint against standard tasks like MMLU or IFEval to measure quality.
- Run recurring evaluations on model endpoints in a CI/CD pipeline to detect performance regressions.
- Compare multiple model endpoints on the same benchmark suite using YAML configs and parallelism settings.
- Evaluate models hosted on NVIDIA's API platform or any OpenAI-compatible endpoint without modifying evaluation code.
- Gather standardized metrics and logs for model selection or release decisions.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Evaluates language models against standardized benchmarks (MMLU, IFEval, MATH, etc.) by sending requests to model API endpoints and collecting results.

Yes, if you need to evaluate language models via API endpoints and want a standardized, well-maintained framework. The large dependency footprint and requirement for a deployed model endpoint are expected trade-offs for comprehensive benchmark support. No known vulnerabilities and active maintenance make it production-ready.

## Install

pip install nvidia-lm-eval
uv add nvidia-lm-eval
poetry add nvidia-lm-eval

## Installing nvidia-lm-eval

Before you install: Low friction install as a pure Python wheel. Actively maintained with recent releases. Brings 34 runtime dependencies including transformers, datasets, and evaluation libraries—a substantial but expected footprint for a comprehensive evaluation framework.

License in practice: MIT license permits commercial and private use with minimal restrictions, making it suitable for integration into proprietary evaluation pipelines.

Quickstart:

pip install nvidia-lm-eval
export MY_API_KEY="your_key"
nemo-evaluator run_eval --eval_type mmlu_pro --model_id my_model --model_url http://localhost:8000 --model_type chat --api_key_name MY_API_KEY --output_dir ./results

Requires a deployed language model endpoint with OpenAI/NIM-compatible API; model must be accessible at the specified URL.

Verify before relying:

- Whether the framework supports evaluation of local models or only remote endpoints.
- Performance characteristics when running large-scale evaluations (throughput, memory usage, parallelism limits).
- Extensibility for custom evaluation tasks beyond the pre-packaged benchmarks.

## Package facts

- License: MIT (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 127.3K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

language model evaluation framework, llm benchmark testing, model endpoint evaluation, standardized llm assessment, evaluation harness for language models, benchmark runner for apis, model quality assessment, llm-evaluation, benchmarking, model-assessment

[View on SkillFed](https://skillfed.io/packages/nvidia-lm-eval) · [View on PyPI](https://pypi.org/project/nvidia-lm-eval/)