pydantic-evals

What it is and what it does

Pydantic Evals is a testing and evaluation framework designed to measure the quality and behavior of stochastic functions—particularly those using LLMs or AI agents. It lets you define test cases with inputs and expected outputs, write custom evaluators to score results, and run batch evaluations with detailed reporting. The library works with any stochastic function implementation, not just Pydantic AI, and includes built-in evaluators for common checks like type validation.

The framework emphasizes type safety and standard Python syntax over domain-specific conventions. It integrates with OpenTelemetry for tracing (with optional Pydantic Logfire integration for visualization), and produces formatted evaluation reports showing scores, assertions, and execution durations. Runtime dependencies are minimal and well-established: pydantic, anyio, pyyaml, rich for output formatting, and logfire-api for tracing.

Use it for:

• Evaluate LLM agent responses against expected outputs using custom scoring logic.
• Run regression tests on stochastic functions to ensure quality across code changes.
• Measure and track evaluation metrics over time with OpenTelemetry tracing to Logfire.
• Define reusable test datasets and evaluators for AI-powered applications.
• Debug function behavior by inspecting full execution traces for each test case.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Pydantic Evals provides a framework for defining test cases, custom evaluators, and running evaluations against stochastic functions—particularly those using LLMs—with built-in support for tracing and result reporting.

Yes. The package is actively maintained, has no known vulnerabilities, uses a permissive MIT license, and solves a real problem for anyone testing LLM-based or stochastic code. Low install friction and standard dependencies make adoption straightforward. Recommended for projects that need structured evaluation of AI functions or non-deterministic behavior.

Install

pydantic-evals on PyPI

pip install pydantic-evals

uv add pydantic-evals

poetry add pydantic-evals

Installing pydantic-evals

Before you install

Low friction installation with a pure-Python wheel. Active maintenance as of 2026-08-14 with no known vulnerabilities. Six runtime dependencies (anyio, logfire-api, pydantic-ai-slim, pydantic, pyyaml, rich) are all standard ecosystem packages.

License in practice

MIT license (permissive) allows use in commercial and private projects with minimal restrictions—only requires attribution and inclusion of the license text.

Quickstart

pip install pydantic-evals

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext

case = Case(name='test', inputs='input', expected_output='output')
dataset = Dataset(name='eval', cases=[case], evaluators=[])
report = dataset.evaluate_sync(async_function)

Requires Python 3.10 or later; async/await syntax expected for task functions.

Verify before relying

• Whether the library's OpenTelemetry integration works with backends other than Pydantic Logfire.
• Performance characteristics when evaluating large datasets or complex functions.
• Whether custom evaluators can be composed or chained beyond the examples shown.

Package facts

Evidence: pydantic_evals-2.30.0-py3-none-any.whl

Tags

More Python Modules packages