---
id: pydantic-evals
version: "2.30.0"
license: MIT
license_treatment: permissive
maintenance: active
---

# pydantic-evals — Framework for evaluating stochastic code execution, especially code making use of LLMs

License: permissive · Maintenance: active · Downloads: 17.5M/mo

## What it is and what it does

Pydantic Evals is a testing and evaluation framework designed to measure the quality and behavior of stochastic functions—particularly those using LLMs or AI agents. It lets you define test cases with inputs and expected outputs, write custom evaluators to score results, and run batch evaluations with detailed reporting. The library works with any stochastic function implementation, not just Pydantic AI, and includes built-in evaluators for common checks like type validation.

The framework emphasizes type safety and standard Python syntax over domain-specific conventions. It integrates with OpenTelemetry for tracing (with optional Pydantic Logfire integration for visualization), and produces formatted evaluation reports showing scores, assertions, and execution durations. Runtime dependencies are minimal and well-established: pydantic, anyio, pyyaml, rich for output formatting, and logfire-api for tracing.

Use it for:

- Evaluate LLM agent responses against expected outputs using custom scoring logic.
- Run regression tests on stochastic functions to ensure quality across code changes.
- Measure and track evaluation metrics over time with OpenTelemetry tracing to Logfire.
- Define reusable test datasets and evaluators for AI-powered applications.
- Debug function behavior by inspecting full execution traces for each test case.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Pydantic Evals provides a framework for defining test cases, custom evaluators, and running evaluations against stochastic functions—particularly those using LLMs—with built-in support for tracing and result reporting.

Yes. The package is actively maintained, has no known vulnerabilities, uses a permissive MIT license, and solves a real problem for anyone testing LLM-based or stochastic code. Low install friction and standard dependencies make adoption straightforward. Recommended for projects that need structured evaluation of AI functions or non-deterministic behavior.

## Install

pip install pydantic-evals
uv add pydantic-evals
poetry add pydantic-evals

## Installing pydantic-evals

Before you install: Low friction installation with a pure-Python wheel. Active maintenance as of 2026-08-14 with no known vulnerabilities. Six runtime dependencies (anyio, logfire-api, pydantic-ai-slim, pydantic, pyyaml, rich) are all standard ecosystem packages.

License in practice: MIT license (permissive) allows use in commercial and private projects with minimal restrictions—only requires attribution and inclusion of the license text.

Quickstart:

pip install pydantic-evals

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext

case = Case(name='test', inputs='input', expected_output='output')
dataset = Dataset(name='eval', cases=[case], evaluators=[])
report = dataset.evaluate_sync(async_function)

Requires Python 3.10 or later; async/await syntax expected for task functions.

Verify before relying:

- Whether the library's OpenTelemetry integration works with backends other than Pydantic Logfire.
- Performance characteristics when evaluating large datasets or complex functions.
- Whether custom evaluators can be composed or chained beyond the examples shown.

## Package facts

- License: MIT (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 17.5M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

evaluate stochastic functions, llm evaluation framework, test case evaluation, ai function testing, custom evaluators, evaluation metrics, stochastic function testing, llm-evaluation, testing-framework, observability

[View on SkillFed](https://skillfed.io/packages/pydantic-evals) · [View on PyPI](https://pypi.org/project/pydantic-evals/)