evidently

What it is and what it does

Evidently is a Python framework for evaluating and monitoring machine learning and LLM systems across the full lifecycle—from offline experiments to live production. It provides Reports for summarizing evaluations (with presets for common tasks like data drift detection and text analysis), Test Suites for adding pass/fail conditions to those reports, and an optional Monitoring Dashboard for tracking metrics over time. The framework works with tabular data, text, and generative outputs, offering both built-in metrics (covering classification, regression, ranking, data quality, and LLM-specific evals) and a Python interface for custom metrics.

The package is modular: you can run one-off evaluations in a notebook or deploy a full monitoring service. It exports results as JSON, HTML, or Python dictionaries and integrates with existing tools through an open architecture. With 26 runtime dependencies—including pandas, scikit-learn, numpy, and plotly—it brings substantial analytical and visualization capabilities but also a large dependency footprint. Active maintenance, permissive licensing, and support for modern Python versions make it production-ready.

Use it for:

• Run data drift detection on production ML models to identify when input distributions shift from training data.
• Evaluate LLM outputs (e.g., RAG systems, summarization) using semantic similarity, retrieval relevance, and custom LLM-as-judge evals.
• Build CI/CD test suites that auto-generate pass/fail conditions from reference datasets to catch model regressions.
• Monitor classification or regression model performance metrics over time via the self-hosted or cloud dashboard.
• Validate data quality in pipelines by checking for missing values, duplicates, new categories, and correlation changes.
• Compare ranking and recommendation system quality using metrics like NDCG, MAP, and diversity scores.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Evidently evaluates, tests, and monitors ML and LLM systems through reports, test suites, and a monitoring dashboard, supporting tabular data, text, and generative tasks with 100+ built-in metrics.

Yes, if you need systematic ML/LLM evaluation and monitoring. The framework is actively maintained, permissively licensed, and offers both lightweight one-off evals and production-grade monitoring. The large dependency footprint and requirement for Python 3.10+ are manageable trade-offs for the breadth of built-in metrics and modular architecture. No known vulnerabilities. Best suited for teams building or operating ML systems that require rigorous testing and observability.

Install

evidently on PyPI

pip install evidently

uv add evidently

poetry add evidently

Installing evidently

Before you install

Low friction install with a pure Python wheel. Active maintenance with recent releases; 7808 repository stars and current support for Python 3.10–3.13 indicate a well-maintained project.

License in practice

Apache License 2.0 (permissive) allows free use, modification, and distribution with minimal restrictions, making it suitable for both open-source and commercial projects.

Quickstart

pip install evidently

import pandas as pd
from evidently import Report
from evidently.presets import DataDriftPreset

report = Report([DataDriftPreset(method="psi")])
my_eval = report.run(reference_data, current_data)
my_eval.save_html("report.html")

Requires Python 3.10 or later; 26 runtime dependencies including pandas, scikit-learn, and numpy mean a substantial environment footprint.

Verify before relying

• Whether the 100+ built-in metrics cover your specific evaluation domain or if custom metric development is required.
• Performance characteristics and scalability limits for large-scale production monitoring workloads.
• Integration complexity with existing ML pipelines and whether the open-source version meets your monitoring SLA needs.

Package facts

Evidence: evidently-0.7.21-py3-none-any.whl

Tags

More Quality Assurance packages

Further reading