autoevals

What it is and what it does

Autoevals is a Python library for automatically grading AI model outputs using a mix of evaluation strategies. It bundles LLM-as-a-judge (where another model scores the output), heuristic methods like Levenshtein distance, and statistical measures like BLEU. The library is designed to let you run evaluations on individual examples, customize prompts, and debug results—useful for assessing factuality, safety, and other subjective qualities of LLM completions.

The package supports both synchronous and asynchronous evaluation, integrates with OpenAI-compatible APIs and the Braintrust Gateway, and allows you to create custom evaluators with your own prompts and output parsing logic. It requires Python 3.10 or higher and depends on four runtime libraries: chevron (templating), jsonschema (validation), polyleven (string distance), and pyyaml (configuration).

Use it for:

• Grade LLM completions for factuality against known correct answers during model development.
• Evaluate safety and content quality of AI-generated text using model-graded scoring.
• Build custom evaluation pipelines with your own prompts and scoring logic for domain-specific tasks.
• Compare model outputs using heuristic metrics like Levenshtein distance or BLEU without calling an external LLM.
• Log and track evaluation results over time using optional Braintrust integration for experiment comparison.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Autoevals provides automatic evaluation methods for AI model outputs, including LLM-as-a-judge, heuristic, and statistical approaches, with built-in support for subjective tasks like fact-checking and safety assessment.

Yes, if you need to evaluate AI model outputs programmatically. The package is actively maintained, has low install friction, and offers a flexible mix of evaluation methods. The main caveat is that license terms are unclear—verify the actual license in the repository before use in proprietary contexts. Requires Python 3.10+ and an LLM API key for most evaluations.

Install

autoevals on PyPI

pip install autoevals

uv add autoevals

poetry add autoevals

Installing autoevals

Before you install

Low install friction with a pure-Python wheel and four lightweight runtime dependencies. Active maintenance with a recent commit on 2026-07-29 and steady popularity (4632526 monthly downloads, top 5000 tier).

License in practice

License treatment is unclear—no SPDX identifier or raw license text is available in the package metadata. Verify the actual license terms in the repository before adopting in proprietary or restricted contexts.

Quickstart

pip install autoevals

from autoevals.llm import Factuality

evaluator = Factuality()
result = evaluator(output="Paris is the capital of France", expected="France's capital", input="Name France's capital")
print(f"Score: {result.score}")

Requires Python 3.10 or higher. Requires OPENAI_API_KEY or BRAINTRUST_API_KEY environment variable set to authenticate with an LLM provider.

Verify before relying

• Whether the package supports evaluation without an external LLM provider or if all evaluations require API calls.
• Specific performance characteristics or latency expectations for typical evaluation workloads.
• Compatibility details with OpenAI SDK versions beyond the stated v0.x and v1.x support.

Package facts

Evidence: autoevals-0.3.0-py3-none-any.whl

Tags

More Artificial Intelligence packages

Further reading