nvidia-nat-eval
Subpackage for evaluation in NVIDIA NeMo Agent Toolkit
Decision gist · record as of 2026-08-14
Yes, if you are building or testing agents within the NVIDIA NeMo Agent Toolkit ecosystem. The package has low install friction, active maintenance, permissive licensing, and no known vulnerabilities. Install only if you need evaluation capabilities; it is a specialized subpackage.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.11 or later (3.11, 3.12, or 3.13 supported)
- Low install friction with a single runtime dependency on nvidia-nat-atif.
- Actively maintained with recent commits and part of a popular toolkit (2576 GitHub stars).
License · maintenance · safety
Apache-2.0 (permissive) — Licensed under Apache-2.0 (permissive), allowing free use, modification, and distribution with minimal restrictions.
last release 2026-06-17 (58 days) · last repo commit 2026-08-12 · 2,576 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 171,435 downloads/mo, #10,366 on PyPI
Alternatives
Verify before relying
pip install nvidia-nat-eval
import nat.plugins.eval- What specific evaluation metrics or methodologies the package implements
- Whether evaluation components work standalone or require other toolkit packages
- What CLI commands are available and their typical use patterns
- How the nat.plugins.eval namespace integrates with broader toolkit workflows
What it is and what it does
nvidia-nat-eval is a subpackage of the NVIDIA NeMo Agent Toolkit that adds evaluation capabilities for testing and assessing agent systems. It provides evaluation-specific components accessible through the nat.plugins.eval namespace, along with command-line tools for running evaluations. The package depends on nvidia-nat-atif as its sole runtime dependency.
This package targets developers building or testing AI agents, particularly those using retrieval-augmented generation patterns. It supports Python 3.11, 3.12, and 3.13, with active maintenance as of August 2026.
Use it for
- Evaluate agent response quality in RAG-based systems during development
- Run CLI-based evaluation workflows as part of agent testing pipelines
- Assess agent behavior and performance metrics during iteration
- Integrate evaluation checks into validation workflows
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you are building or testing agents within the NVIDIA NeMo Agent Toolkit ecosystem.
The package has low install friction, active maintenance, permissive licensing, and no known vulnerabilities. Install only if you need evaluation capabilities; it is a specialized subpackage.
Install
nvidia-nat-eval on PyPI
Before you install
Low install friction with a single runtime dependency on nvidia-nat-atif. Actively maintained with recent commits and part of a popular toolkit (2576 GitHub stars).
Requires Python 3.11 or later (3.11, 3.12, or 3.13 supported)
License in practice
Licensed under Apache-2.0 (permissive), allowing free use, modification, and distribution with minimal restrictions.
Quickstart
pip install nvidia-nat-eval
import nat.plugins.evalVerify before relying
- What specific evaluation metrics or methodologies the package implements
- Whether evaluation components work standalone or require other toolkit packages
- What CLI commands are available and their typical use patterns
- How the nat.plugins.eval namespace integrates with broader toolkit workflows
Package facts
| License | Apache-2.0 permissive |
| Python support | Supports the current Python release <3.14,>=3.11 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 1 packagenvidia-nat-atif |
| Maintenance | Actively maintained 58 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 171,435 / month, #10,366 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Programming Language :: PythonProgramming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13 |
Evidence: nvidia_nat_eval-1.8.0-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “agent evaluation framework”
- nvidia-nat-evalProvides evaluation components and CLI commands for the NVIDIA NeMo…
- agent-framework-labProvides experimental lab modules (GAIA, TAU2, Lightning) built on…
- nvidia-nat-atifProvides ATIF schema models for the NVIDIA NeMo Agent Toolkit,…
Give your agent the search over MCP, or paste the wish link into any chat.
More Artificial Intelligence packages
LiteLLM provides a unified Python interface to call 100+ LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Azure, and others) using OpenAI-compatible API format, available as both a Python SDK and a self-hosted AI Gateway proxy server.
Install it if you need to work with multiple LLM providers or want to centralize LLM routing in your organization.
Client library and CLI tool for downloading, uploading, and managing models, datasets, and repositories on the Hugging Face Hub platform.
Install it if you work with Hugging Face Hub models or datasets.
LangChain provides a framework for building agents and LLM-powered applications by composing language models, tools, and memory through a unified API that abstracts over multiple model providers.
hf-xet provides chunk-based deduplication and efficient file transfer for the Hugging Face Hub, enabling faster uploads and downloads of large files with local disk caching.
Tokenizers converts raw text into token sequences for NLP models, with support for training custom vocabularies and using pre-built tokenizers (BPE, WordPiece) optimized for speed via Rust.
Transformers provides a unified framework for loading, fine-tuning, and running state-of-the-art pretrained models across text, vision, audio, video, and multimodal tasks using PyTorch, JAX, or TensorFlow.
Install it if you need to run or train any transformer-based model for NLP, vision, audio, or multimodal tasks.
See also nv-ingest-client · nvidia-nat-atif · nvidia-nat-langchain · nvidia-nat-opentelemetry · nvidia-nat-core · nvidia-nat-mcp · nemo-relay · nvidia-lm-eval · nvidia-nat · nemo-evaluator