autoevals
Universal library for evaluating AI models
Decision gist · record as of 2026-08-14
Yes, if you need to evaluate AI model outputs programmatically. The package is actively maintained, has low install friction, and offers a flexible mix of evaluation methods. The main caveat is that license terms are unclear—verify the actual license in the repository before use in proprietary contexts. Requires Python 3.10+ and an LLM API key for most evaluations.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.10 or higher.
- Requires OPENAI_API_KEY or BRAINTRUST_API_KEY environment variable set to authenticate with an LLM provider.
- Low install friction with a pure-Python wheel and four lightweight runtime dependencies.
License · maintenance · safety
(unclear) — License treatment is unclear—no SPDX identifier or raw license text is available in the package metadata. Verify the actual license terms in the repository before adopting in proprietary or restricted contexts.
last release 2026-06-09 (66 days) · last repo commit 2026-07-29 · 999 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 4,632,526 downloads/mo, #2,270 on PyPI
Alternatives
Verify before relying
pip install autoevals
from autoevals.llm import Factuality
evaluator = Factuality()
result = evaluator(output="Paris is the capital of France", expected="France's capital", input="Name France's capital")
print(f"Score: {result.score}")- Whether the package supports evaluation without an external LLM provider or if all evaluations require API calls.
- Specific performance characteristics or latency expectations for typical evaluation workloads.
- Compatibility details with OpenAI SDK versions beyond the stated v0.x and v1.x support.
What it is and what it does
Autoevals is a Python library for automatically grading AI model outputs using a mix of evaluation strategies. It bundles LLM-as-a-judge (where another model scores the output), heuristic methods like Levenshtein distance, and statistical measures like BLEU. The library is designed to let you run evaluations on individual examples, customize prompts, and debug results—useful for assessing factuality, safety, and other subjective qualities of LLM completions.
The package supports both synchronous and asynchronous evaluation, integrates with OpenAI-compatible APIs and the Braintrust Gateway, and allows you to create custom evaluators with your own prompts and output parsing logic. It requires Python 3.10 or higher and depends on four runtime libraries: chevron (templating), jsonschema (validation), polyleven (string distance), and pyyaml (configuration).
Use it for
- Grade LLM completions for factuality against known correct answers during model development.
- Evaluate safety and content quality of AI-generated text using model-graded scoring.
- Build custom evaluation pipelines with your own prompts and scoring logic for domain-specific tasks.
- Compare model outputs using heuristic metrics like Levenshtein distance or BLEU without calling an external LLM.
- Log and track evaluation results over time using optional Braintrust integration for experiment comparison.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you need to evaluate AI model outputs programmatically.
The package is actively maintained, has low install friction, and offers a flexible mix of evaluation methods. The main caveat is that license terms are unclear—verify the actual license in the repository before use in proprietary contexts. Requires Python 3.10+ and an LLM API key for most evaluations.
Install
autoevals on PyPI
Before you install
Low install friction with a pure-Python wheel and four lightweight runtime dependencies. Active maintenance with a recent commit on 2026-07-29 and steady popularity (4632526 monthly downloads, top 5000 tier).
Requires Python 3.10 or higher. Requires OPENAI_API_KEY or BRAINTRUST_API_KEY environment variable set to authenticate with an LLM provider.
License in practice
License treatment is unclear—no SPDX identifier or raw license text is available in the package metadata. Verify the actual license terms in the repository before adopting in proprietary or restricted contexts.
Quickstart
pip install autoevals
from autoevals.llm import Factuality
evaluator = Factuality()
result = evaluator(output="Paris is the capital of France", expected="France's capital", input="Name France's capital")
print(f"Score: {result.score}")Verify before relying
- Whether the package supports evaluation without an external LLM provider or if all evaluations require API calls.
- Specific performance characteristics or latency expectations for typical evaluation workloads.
- Compatibility details with OpenAI SDK versions beyond the stated v0.x and v1.x support.
Package facts
| License | Not declared unclear |
| Python support | Supports the current Python release >=3.10.0 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 4 packageschevronjsonschemapolylevenpyyaml |
| Maintenance | Actively maintained 66 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 4,632,526 / month, #2,270 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Operating System :: OS IndependentProgramming Language :: Python :: 3Programming Language :: Python :: 3.10 |
Evidence: autoevals-0.3.0-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “automatic evaluation framework”
- autoevalsAutoevals provides automatic evaluation methods for AI model outputs,…
- rouge-metricComputes ROUGE metrics (ROUGE-N, ROUGE-L, ROUGE-W, ROUGE-S, ROUGE-SU)…
- lmnrPython SDK that traces, evaluates, and analyzes LLM application…
Give your agent the search over MCP, or paste the wish link into any chat.
More Artificial Intelligence packages
LiteLLM provides a unified Python interface to call 100+ LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Azure, and others) using OpenAI-compatible API format, available as both a Python SDK and a self-hosted AI Gateway proxy server.
Install it if you need to work with multiple LLM providers or want to centralize LLM routing in your organization.
Client library and CLI tool for downloading, uploading, and managing models, datasets, and repositories on the Hugging Face Hub platform.
Install it if you work with Hugging Face Hub models or datasets.
LangChain provides a framework for building agents and LLM-powered applications by composing language models, tools, and memory through a unified API that abstracts over multiple model providers.
hf-xet provides chunk-based deduplication and efficient file transfer for the Hugging Face Hub, enabling faster uploads and downloads of large files with local disk caching.
Tokenizers converts raw text into token sequences for NLP models, with support for training custom vocabularies and using pre-built tokenizers (BPE, WordPiece) optimized for speed via Rust.
Transformers provides a unified framework for loading, fine-tuning, and running state-of-the-art pretrained models across text, vision, audio, video, and multimodal tasks using PyTorch, JAX, or TensorFlow.
Install it if you need to run or train any transformer-based model for NLP, vision, audio, or multimodal tasks.
See also openevals · arize-phoenix-evals · deepeval · azure-ai-evaluation · agentevals · inspect-ai · strands-agents-evals · inspect-evals · pydantic-evals · rubric