deepeval

What it is and what it does

DeepEval is a pytest-like testing framework designed specifically for evaluating large language model applications. It provides a collection of pre-built metrics—including G-Eval, answer relevancy, faithfulness, hallucination detection, and agentic metrics—that measure the quality of LLM outputs against criteria like factual accuracy, relevance, and task completion. The framework supports end-to-end evaluation of black-box LLM systems, component-level testing of individual steps (retrieval, tool use, agent handoffs), and multi-turn conversation assessment.

The package integrates with any LLM framework (OpenAI, LangChain, Claude) and CI/CD pipeline. Many of its metrics run locally on your machine using NLP models or statistical methods, though some can also delegate to external LLMs for judgment. It includes tools for synthetic dataset generation, prompt optimization, and benchmarking against standard LLM benchmarks. With 28 runtime dependencies covering async HTTP, CLI tooling, observability (OpenTelemetry, PostHog), and testing utilities, it's designed as a comprehensive evaluation platform rather than a minimal library.

Use it for:

• Evaluate RAG pipeline output quality by measuring answer relevancy, faithfulness, and retrieval context precision.
• Test AI agent behavior across decision trees by checking task completion, tool correctness, and plan adherence.
• Monitor chatbot consistency and factual grounding across multi-turn conversations using turn-level metrics.
• Detect hallucinations and bias in LLM outputs before deploying to production.
• Compare model performance (OpenAI vs. Claude) or prompt variations using standardized metrics.
• Benchmark your LLM against public benchmarks like MMLU or HumanEval in minimal code.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

DeepEval is an LLM evaluation framework that runs unit tests on language model applications using metrics like G-Eval, answer relevancy, and hallucination detection, with many metrics executing locally on your machine.

Yes. DeepEval is actively maintained, has no known vulnerabilities, installs with low friction, and is permissively licensed. It's well-suited for teams building LLM applications who need systematic evaluation beyond manual testing. The large dependency footprint and reliance on external LLM APIs for some metrics are trade-offs for comprehensive evaluation coverage; verify that the specific metrics you need match your local-vs.-API execution preferences before committing.

Install

deepeval on PyPI

pip install deepeval

uv add deepeval

poetry add deepeval

Installing deepeval

Before you install

Low friction install with a pure Python wheel. Active maintenance with recent releases (2 days old) and strong community signal (17597 stars). Supports Python 3.9 through 3.14.

License in practice

Apache-2.0 permissive license allows commercial use, modification, and distribution with minimal restrictions—suitable for most production and proprietary projects.

Quickstart

pip install deepeval

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancy

metric = AnswerRelevancy()
result = metric.measure(prediction="answer", input="question")
print(result.score)

Requires an LLM provider (OpenAI, Claude, etc.) or local NLP models to run metrics; some metrics depend on external API calls unless configured for local execution.

Verify before relying

• Whether all 28 runtime dependencies are required for basic use or if many are optional (testing, CLI, integrations).
• Performance characteristics and latency for running metrics on large evaluation batches.
• Which metrics actually run locally versus requiring external LLM API calls by default.

Package facts

Evidence: deepeval-4.1.8-py3-none-any.whl

Tags

More Testing packages

Further reading