pydantic-evals
Framework for evaluating stochastic code execution, especially code making use of LLMs
Decision gist · record as of 2026-08-14
Yes. The package is actively maintained, has no known vulnerabilities, uses a permissive MIT license, and solves a real problem for anyone testing LLM-based or stochastic code. Low install friction and standard dependencies make adoption straightforward. Recommended for projects that need structured evaluation of AI functions or non-deterministic behavior.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.10 or later; async/await syntax expected for task functions.
- Low friction installation with a pure-Python wheel.
- Active maintenance as of 2026-08-14 with no known vulnerabilities.
License · maintenance · safety
MIT (permissive) — MIT license (permissive) allows use in commercial and private projects with minimal restrictions—only requires attribution and inclusion of the license text.
last release 2026-08-14 (0 days) · last repo commit 2026-08-14 · 19,295 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 17,486,604 downloads/mo, #1,118 on PyPI
Alternatives
Verify before relying
pip install pydantic-evals
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext
case = Case(name='test', inputs='input', expected_output='output')
dataset = Dataset(name='eval', cases=[case], evaluators=[])
report = dataset.evaluate_sync(async_function)- Whether the library's OpenTelemetry integration works with backends other than Pydantic Logfire.
- Performance characteristics when evaluating large datasets or complex functions.
- Whether custom evaluators can be composed or chained beyond the examples shown.
What it is and what it does
Pydantic Evals is a testing and evaluation framework designed to measure the quality and behavior of stochastic functions—particularly those using LLMs or AI agents. It lets you define test cases with inputs and expected outputs, write custom evaluators to score results, and run batch evaluations with detailed reporting. The library works with any stochastic function implementation, not just Pydantic AI, and includes built-in evaluators for common checks like type validation.
The framework emphasizes type safety and standard Python syntax over domain-specific conventions. It integrates with OpenTelemetry for tracing (with optional Pydantic Logfire integration for visualization), and produces formatted evaluation reports showing scores, assertions, and execution durations. Runtime dependencies are minimal and well-established: pydantic, anyio, pyyaml, rich for output formatting, and logfire-api for tracing.
Use it for
- Evaluate LLM agent responses against expected outputs using custom scoring logic.
- Run regression tests on stochastic functions to ensure quality across code changes.
- Measure and track evaluation metrics over time with OpenTelemetry tracing to Logfire.
- Define reusable test datasets and evaluators for AI-powered applications.
- Debug function behavior by inspecting full execution traces for each test case.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
The package is actively maintained, has no known vulnerabilities, uses a permissive MIT license, and solves a real problem for anyone testing LLM-based or stochastic code. Low install friction and standard dependencies make adoption straightforward. Recommended for projects that need structured evaluation of AI functions or non-deterministic behavior.
Install
pydantic-evals on PyPI
Before you install
Low friction installation with a pure-Python wheel. Active maintenance as of 2026-08-14 with no known vulnerabilities. Six runtime dependencies (anyio, logfire-api, pydantic-ai-slim, pydantic, pyyaml, rich) are all standard ecosystem packages.
Requires Python 3.10 or later; async/await syntax expected for task functions.
License in practice
MIT license (permissive) allows use in commercial and private projects with minimal restrictions—only requires attribution and inclusion of the license text.
Quickstart
pip install pydantic-evals
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext
case = Case(name='test', inputs='input', expected_output='output')
dataset = Dataset(name='eval', cases=[case], evaluators=[])
report = dataset.evaluate_sync(async_function)Verify before relying
- Whether the library's OpenTelemetry integration works with backends other than Pydantic Logfire.
- Performance characteristics when evaluating large datasets or complex functions.
- Whether custom evaluators can be composed or chained beyond the examples shown.
Package facts
| License | MIT permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 6 packagesanyiologfire-apipydantic-ai-slimpydanticpyyamlrich |
| Maintenance | Actively maintained 0 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 17,486,604 / month, #1,118 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableEnvironment :: ConsoleEnvironment :: MacOS XIntended Audience :: DevelopersIntended Audience :: Information TechnologyIntended Audience :: System AdministratorsLicense :: OSI Approved :: MIT LicenseOperating System :: POSIX :: LinuxOperating System :: UnixProgramming Language :: PythonProgramming Language :: Python :: 3Programming Language :: Python :: 3 :: OnlyProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Topic :: InternetTopic :: Software Development :: Libraries :: Python Modules |
Evidence: pydantic_evals-2.30.0-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “evaluate stochastic functions”
- pydantic-evalsPydantic Evals provides a framework for defining test cases, custom…
- moocoreProvides fast implementations of core algorithms for multi-objective…
- torchsdeSolves stochastic differential equations (SDEs) with GPU support and…
Give your agent the search over MCP, or paste the wish link into any chat.
More Python Modules packages
Converts domain names between Unicode and ASCII-compatible encoding (Punycode) according to IDNA 2008 and Unicode Technical Standard 46, with security validation and broader script coverage than the standard library.
Install it if you work with internationalized domain names, need to validate domains, or use HTTP clients that depend on it transitively.
Setuptools is a Python build backend and package management tool that handles building, distributing, and installing Python packages, including support for C/C++ extension modules.
PyYAML parses and emits YAML 1.1 data format, enabling serialization and deserialization of configuration files and Python objects to and from human-readable YAML text.
Pydantic validates Python data structures against type hints, coercing and checking input at runtime to ensure it matches a declared schema.
Provides reusable metadata objects for use with PEP-593 `typing.Annotated` to express common constraints like bounds, collection sizes, and predicates on types.
Install it if you use or build libraries that need to express type constraints in a standardized, inspectable way—or if you want to annotate your own types with…
Provides runtime tools to inspect and introspect Python type annotations, enabling programmatic examination of type hints at execution time.
See also pydantic-ai · arize-phoenix-evals · openevals · autoevals · agentevals · azure-ai-evaluation · pydantic-monty · pydantic-graph · strands-agents-evals · py-expression-eval