{"categories":[{"label":"Testing","url":"https://skillfed.io/packages/category/software-development-testing"}],"enrichment":{"capability":"DeepEval is an LLM evaluation framework that runs unit tests on language model applications using metrics like G-Eval, answer relevancy, and hallucination detection, with many metrics executing locally on your machine.","skillfed_tags":["llm-evaluation","ai-testing","rag-metrics"],"use_cases":["Evaluate RAG pipeline output quality by measuring answer relevancy, faithfulness, and retrieval context precision.","Test AI agent behavior across decision trees by checking task completion, tool correctness, and plan adherence.","Monitor chatbot consistency and factual grounding across multi-turn conversations using turn-level metrics.","Detect hallucinations and bias in LLM outputs before deploying to production.","Compare model performance (OpenAI vs. Claude) or prompt variations using standardized metrics.","Benchmark your LLM against public benchmarks like MMLU or HumanEval in minimal code."],"what_it_does":"DeepEval is a pytest-like testing framework designed specifically for evaluating large language model applications. It provides a collection of pre-built metrics\u2014including G-Eval, answer relevancy, faithfulness, hallucination detection, and agentic metrics\u2014that measure the quality of LLM outputs against criteria like factual accuracy, relevance, and task completion. The framework supports end-to-end evaluation of black-box LLM systems, component-level testing of individual steps (retrieval, tool use, agent handoffs), and multi-turn conversation assessment.\n\nThe package integrates with any LLM framework (OpenAI, LangChain, Claude) and CI/CD pipeline. Many of its metrics run locally on your machine using NLP models or statistical methods, though some can also delegate to external LLMs for judgment. It includes tools for synthetic dataset generation, prompt optimization, and benchmarking against standard LLM benchmarks. With 28 runtime dependencies covering async HTTP, CLI tooling, observability (OpenTelemetry, PostHog), and testing utilities, it's designed as a comprehensive evaluation platform rather than a minimal library.","worth_installing":"Yes. DeepEval is actively maintained, has no known vulnerabilities, installs with low friction, and is permissively licensed. It's well-suited for teams building LLM applications who need systematic evaluation beyond manual testing. The large dependency footprint and reliance on external LLM APIs for some metrics are trade-offs for comprehensive evaluation coverage; verify that the specific metrics you need match your local-vs.-API execution preferences before committing."},"id":"deepeval","links":{"html":"https://skillfed.io/packages/deepeval","md":"https://skillfed.io/packages/deepeval.md","pypi":"https://pypi.org/project/deepeval/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-12","license_spdx":null,"license_treatment":"permissive","name":"deepeval","python_support":"supports_current","summary":"The LLM Evaluation Framework"},"popularity":{"monthly_downloads":6466639,"position":1906,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"4.1.8"}