nvidia-lm-eval
A framework for evaluating language models - packaged by NVIDIA
Decision gist · record as of 2026-08-14
Yes, if you need to evaluate language models via API endpoints and want a standardized, well-maintained framework. The large dependency footprint and requirement for a deployed model endpoint are expected trade-offs for comprehensive benchmark support. No known vulnerabilities and active maintenance make it production-ready.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires a deployed language model endpoint with OpenAI/NIM-compatible API; model must be accessible at the specified URL.
- Low friction install as a pure Python wheel.
- Actively maintained with recent releases.
License · maintenance · safety
MIT (permissive) — MIT license permits commercial and private use with minimal restrictions, making it suitable for integration into proprietary evaluation pipelines.
last release 2026-03-16 (151 days)
0 known vulnerabilities (OSV.dev, 2026-08-14) · 127,294 downloads/mo, #11,748 on PyPI
Alternatives
Verify before relying
pip install nvidia-lm-eval
export MY_API_KEY="your_key"
nemo-evaluator run_eval --eval_type mmlu_pro --model_id my_model --model_url http://localhost:8000 --model_type chat --api_key_name MY_API_KEY --output_dir ./results- Whether the framework supports evaluation of local models or only remote endpoints.
- Performance characteristics when running large-scale evaluations (throughput, memory usage, parallelism limits).
- Extensibility for custom evaluation tasks beyond the pre-packaged benchmarks.
What it is and what it does
NVIDIA NeMo Evaluator is a framework for running standardized benchmarks against language model endpoints. It wraps the lm-evaluation-harness with NVIDIA's tooling to simplify evaluation workflows, providing a command-line interface and YAML-based configuration system. You point it at a model API (chat or completions style), specify a benchmark task (MMLU, IFEval, MATH, etc.), and it handles request batching, result collection, and output formatting.
The package is designed for teams evaluating models deployed as API endpoints rather than local inference. It depends on a large ecosystem of evaluation libraries (evaluate, datasets, rouge-score, sacrebleu, transformers) and integrates with OpenAI-compatible APIs. Configuration is hierarchical—command-line flags override YAML config, which overrides task defaults. Results are saved to a specified directory as YAML.
Use it for
- Benchmark a custom or third-party LLM endpoint against standard tasks like MMLU or IFEval to measure quality.
- Run recurring evaluations on model endpoints in a CI/CD pipeline to detect performance regressions.
- Compare multiple model endpoints on the same benchmark suite using YAML configs and parallelism settings.
- Evaluate models hosted on NVIDIA's API platform or any OpenAI-compatible endpoint without modifying evaluation code.
- Gather standardized metrics and logs for model selection or release decisions.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you need to evaluate language models via API endpoints and want a standardized, well-maintained framework.
The large dependency footprint and requirement for a deployed model endpoint are expected trade-offs for comprehensive benchmark support. No known vulnerabilities and active maintenance make it production-ready.
Install
nvidia-lm-eval on PyPI
Before you install
Low friction install as a pure Python wheel. Actively maintained with recent releases. Brings 34 runtime dependencies including transformers, datasets, and evaluation libraries—a substantial but expected footprint for a comprehensive evaluation framework.
Requires a deployed language model endpoint with OpenAI/NIM-compatible API; model must be accessible at the specified URL.
License in practice
MIT license permits commercial and private use with minimal restrictions, making it suitable for integration into proprietary evaluation pipelines.
Quickstart
pip install nvidia-lm-eval
export MY_API_KEY="your_key"
nemo-evaluator run_eval --eval_type mmlu_pro --model_id my_model --model_url http://localhost:8000 --model_type chat --api_key_name MY_API_KEY --output_dir ./resultsVerify before relying
- Whether the framework supports evaluation of local models or only remote endpoints.
- Performance characteristics when running large-scale evaluations (throughput, memory usage, parallelism limits).
- Extensibility for custom evaluation tasks beyond the pre-packaged benchmarks.
Package facts
| License | MIT permissive |
| Python support | Supports the current Python release >=3.9 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 34 packagesevaluatedatasetsjsonlinesnumexprpybind11pytablewriterrouge-scoresacrebleuscikit-learntqdm-multiprocesstransformerszstandarddillword2numbermore_itertoolspandasjinja2numpyhttpxlangdetectimmutabledictnltkrequestsaiohttptenacitytqdmtiktokensentencepiecebeautifulsoup4openai |
| Maintenance | Actively maintained 151 days since the last release |
| First released | |
| Downloads | 127,294 / month, #11,748 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableLicense :: OSI Approved :: MIT LicenseOperating System :: OS IndependentProgramming Language :: Python :: 3 |
Evidence: nvidia_lm_eval-26.3-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “model endpoint evaluation”
- nvidia-lm-evalEvaluates language models against standardized benchmarks (MMLU,…
- arize-phoenixPhoenix is an open-source AI observability platform that traces LLM…
- garakGarak is a command-line tool that probes large language models for…
Give your agent the search over MCP, or paste the wish link into any chat.
More Artificial Intelligence packages
LiteLLM provides a unified Python interface to call 100+ LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Azure, and others) using OpenAI-compatible API format, available as both a Python SDK and a self-hosted AI Gateway proxy server.
Install it if you need to work with multiple LLM providers or want to centralize LLM routing in your organization.
Client library and CLI tool for downloading, uploading, and managing models, datasets, and repositories on the Hugging Face Hub platform.
Install it if you work with Hugging Face Hub models or datasets.
LangChain provides a framework for building agents and LLM-powered applications by composing language models, tools, and memory through a unified API that abstracts over multiple model providers.
hf-xet provides chunk-based deduplication and efficient file transfer for the Hugging Face Hub, enabling faster uploads and downloads of large files with local disk caching.
Tokenizers converts raw text into token sequences for NLP models, with support for training custom vocabularies and using pre-built tokenizers (BPE, WordPiece) optimized for speed via Rust.
Transformers provides a unified framework for loading, fine-tuning, and running state-of-the-art pretrained models across text, vision, audio, video, and multimodal tasks using PyTorch, JAX, or TensorFlow.
Install it if you need to run or train any transformer-based model for NLP, vision, audio, or multimodal tasks.
See also lm-eval · nemo-evaluator · nvidia-nat-eval · learnosity-sdk · autoevals · nvidia-nat-core · openevals · azure-ai-evaluation · nemo-gym · nvidia-nat-atif