{"categories":[{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/2"}],"enrichment":{"capability":"Autoevals provides automatic evaluation methods for AI model outputs, including LLM-as-a-judge, heuristic, and statistical approaches, with built-in support for subjective tasks like fact-checking and safety assessment.","skillfed_tags":["llm-evaluation","model-grading","ai-testing"],"use_cases":["Grade LLM completions for factuality against known correct answers during model development.","Evaluate safety and content quality of AI-generated text using model-graded scoring.","Build custom evaluation pipelines with your own prompts and scoring logic for domain-specific tasks.","Compare model outputs using heuristic metrics like Levenshtein distance or BLEU without calling an external LLM.","Log and track evaluation results over time using optional Braintrust integration for experiment comparison."],"what_it_does":"Autoevals is a Python library for automatically grading AI model outputs using a mix of evaluation strategies. It bundles LLM-as-a-judge (where another model scores the output), heuristic methods like Levenshtein distance, and statistical measures like BLEU. The library is designed to let you run evaluations on individual examples, customize prompts, and debug results\u2014useful for assessing factuality, safety, and other subjective qualities of LLM completions.\n\nThe package supports both synchronous and asynchronous evaluation, integrates with OpenAI-compatible APIs and the Braintrust Gateway, and allows you to create custom evaluators with your own prompts and output parsing logic. It requires Python 3.10 or higher and depends on four runtime libraries: chevron (templating), jsonschema (validation), polyleven (string distance), and pyyaml (configuration).","worth_installing":"Yes, if you need to evaluate AI model outputs programmatically. The package is actively maintained, has low install friction, and offers a flexible mix of evaluation methods. The main caveat is that license terms are unclear\u2014verify the actual license in the repository before use in proprietary contexts. Requires Python 3.10+ and an LLM API key for most evaluations."},"id":"autoevals","links":{"html":"https://skillfed.io/packages/autoevals","md":"https://skillfed.io/packages/autoevals.md","pypi":"https://pypi.org/project/autoevals/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-06-09","license_spdx":null,"license_treatment":"unclear","name":"autoevals","python_support":"supports_current","summary":"Universal library for evaluating AI models"},"popularity":{"monthly_downloads":4632526,"position":2270,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"0.3.0"}