{"categories":[{"label":"Python Modules","url":"https://skillfed.io/packages/category/software-development-libraries-python-modules/3"},{"label":"Internet","url":"https://skillfed.io/packages/category/internet"}],"enrichment":{"capability":"Pydantic Evals provides a framework for defining test cases, custom evaluators, and running evaluations against stochastic functions\u2014particularly those using LLMs\u2014with built-in support for tracing and result reporting.","skillfed_tags":["llm-evaluation","testing-framework","observability"],"use_cases":["Evaluate LLM agent responses against expected outputs using custom scoring logic.","Run regression tests on stochastic functions to ensure quality across code changes.","Measure and track evaluation metrics over time with OpenTelemetry tracing to Logfire.","Define reusable test datasets and evaluators for AI-powered applications.","Debug function behavior by inspecting full execution traces for each test case."],"what_it_does":"Pydantic Evals is a testing and evaluation framework designed to measure the quality and behavior of stochastic functions\u2014particularly those using LLMs or AI agents. It lets you define test cases with inputs and expected outputs, write custom evaluators to score results, and run batch evaluations with detailed reporting. The library works with any stochastic function implementation, not just Pydantic AI, and includes built-in evaluators for common checks like type validation.\n\nThe framework emphasizes type safety and standard Python syntax over domain-specific conventions. It integrates with OpenTelemetry for tracing (with optional Pydantic Logfire integration for visualization), and produces formatted evaluation reports showing scores, assertions, and execution durations. Runtime dependencies are minimal and well-established: pydantic, anyio, pyyaml, rich for output formatting, and logfire-api for tracing.","worth_installing":"Yes. The package is actively maintained, has no known vulnerabilities, uses a permissive MIT license, and solves a real problem for anyone testing LLM-based or stochastic code. Low install friction and standard dependencies make adoption straightforward. Recommended for projects that need structured evaluation of AI functions or non-deterministic behavior."},"id":"pydantic-evals","links":{"html":"https://skillfed.io/packages/pydantic-evals","md":"https://skillfed.io/packages/pydantic-evals.md","pypi":"https://pypi.org/project/pydantic-evals/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-14","license_spdx":"MIT","license_treatment":"permissive","name":"pydantic-evals","python_support":"supports_current","summary":"Framework for evaluating stochastic code execution, especially code making use of LLMs"},"popularity":{"monthly_downloads":17486604,"position":1118,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"2.30.0"}