nvidia-lm-eval

What it is and what it does

NVIDIA NeMo Evaluator is a framework for running standardized benchmarks against language model endpoints. It wraps the lm-evaluation-harness with NVIDIA's tooling to simplify evaluation workflows, providing a command-line interface and YAML-based configuration system. You point it at a model API (chat or completions style), specify a benchmark task (MMLU, IFEval, MATH, etc.), and it handles request batching, result collection, and output formatting.

The package is designed for teams evaluating models deployed as API endpoints rather than local inference. It depends on a large ecosystem of evaluation libraries (evaluate, datasets, rouge-score, sacrebleu, transformers) and integrates with OpenAI-compatible APIs. Configuration is hierarchical—command-line flags override YAML config, which overrides task defaults. Results are saved to a specified directory as YAML.

Use it for:

• Benchmark a custom or third-party LLM endpoint against standard tasks like MMLU or IFEval to measure quality.
• Run recurring evaluations on model endpoints in a CI/CD pipeline to detect performance regressions.
• Compare multiple model endpoints on the same benchmark suite using YAML configs and parallelism settings.
• Evaluate models hosted on NVIDIA's API platform or any OpenAI-compatible endpoint without modifying evaluation code.
• Gather standardized metrics and logs for model selection or release decisions.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Evaluates language models against standardized benchmarks (MMLU, IFEval, MATH, etc.) by sending requests to model API endpoints and collecting results.

Yes, if you need to evaluate language models via API endpoints and want a standardized, well-maintained framework. The large dependency footprint and requirement for a deployed model endpoint are expected trade-offs for comprehensive benchmark support. No known vulnerabilities and active maintenance make it production-ready.

Install

nvidia-lm-eval on PyPI

pip install nvidia-lm-eval

uv add nvidia-lm-eval

poetry add nvidia-lm-eval

Installing nvidia-lm-eval

Before you install

Low friction install as a pure Python wheel. Actively maintained with recent releases. Brings 34 runtime dependencies including transformers, datasets, and evaluation libraries—a substantial but expected footprint for a comprehensive evaluation framework.

License in practice

MIT license permits commercial and private use with minimal restrictions, making it suitable for integration into proprietary evaluation pipelines.

Quickstart

pip install nvidia-lm-eval
export MY_API_KEY="your_key"
nemo-evaluator run_eval --eval_type mmlu_pro --model_id my_model --model_url http://localhost:8000 --model_type chat --api_key_name MY_API_KEY --output_dir ./results

Requires a deployed language model endpoint with OpenAI/NIM-compatible API; model must be accessible at the specified URL.

Verify before relying

• Whether the framework supports evaluation of local models or only remote endpoints.
• Performance characteristics when running large-scale evaluations (throughput, memory usage, parallelism limits).
• Extensibility for custom evaluation tasks beyond the pre-packaged benchmarks.

Package facts

Evidence: nvidia_lm_eval-26.3-py3-none-any.whl

Tags

More Artificial Intelligence packages

Further reading