{"categories":[{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/9"}],"enrichment":{"capability":"Evaluates language models against standardized benchmarks (MMLU, IFEval, MATH, etc.) by sending requests to model API endpoints and collecting results.","skillfed_tags":["llm-evaluation","benchmarking","model-assessment"],"use_cases":["Benchmark a custom or third-party LLM endpoint against standard tasks like MMLU or IFEval to measure quality.","Run recurring evaluations on model endpoints in a CI/CD pipeline to detect performance regressions.","Compare multiple model endpoints on the same benchmark suite using YAML configs and parallelism settings.","Evaluate models hosted on NVIDIA's API platform or any OpenAI-compatible endpoint without modifying evaluation code.","Gather standardized metrics and logs for model selection or release decisions."],"what_it_does":"NVIDIA NeMo Evaluator is a framework for running standardized benchmarks against language model endpoints. It wraps the lm-evaluation-harness with NVIDIA's tooling to simplify evaluation workflows, providing a command-line interface and YAML-based configuration system. You point it at a model API (chat or completions style), specify a benchmark task (MMLU, IFEval, MATH, etc.), and it handles request batching, result collection, and output formatting.\n\nThe package is designed for teams evaluating models deployed as API endpoints rather than local inference. It depends on a large ecosystem of evaluation libraries (evaluate, datasets, rouge-score, sacrebleu, transformers) and integrates with OpenAI-compatible APIs. Configuration is hierarchical\u2014command-line flags override YAML config, which overrides task defaults. Results are saved to a specified directory as YAML.","worth_installing":"Yes, if you need to evaluate language models via API endpoints and want a standardized, well-maintained framework. The large dependency footprint and requirement for a deployed model endpoint are expected trade-offs for comprehensive benchmark support. No known vulnerabilities and active maintenance make it production-ready."},"id":"nvidia-lm-eval","links":{"html":"https://skillfed.io/packages/nvidia-lm-eval","md":"https://skillfed.io/packages/nvidia-lm-eval.md","pypi":"https://pypi.org/project/nvidia-lm-eval/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-03-16","license_spdx":null,"license_treatment":"permissive","name":"nvidia-lm-eval","python_support":"supports_current","summary":"A framework for evaluating language models - packaged by NVIDIA"},"popularity":{"monthly_downloads":127294,"position":11748,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"26.3"}