pydeequ

What it is and what it does

PyDeequ wraps Deequ, an AWS-built library on Apache Spark, to define and run data quality checks at scale. It lets you compute metrics on dataset columns (via Analyzers and Profiles), suggest validation constraints based on data patterns, verify datasets against those constraints, and persist quality metrics over time in a repository. The package is designed for teams building data pipelines who need to catch data quality issues before they propagate downstream—treating data validation like unit tests for code.

You define checks declaratively (e.g., "column X must be complete and unique"), run them across large DataFrames, and get back pass/fail results at both aggregate and row levels. The row-level output lets you quarantine problematic records. It integrates with numpy and pandas as runtime dependencies.

Use it for:

• Run data profiling on large datasets to compute completeness, uniqueness, and distribution metrics before loading into a data warehouse.
• Define and verify data quality constraints on incoming data in a data pipeline to catch schema violations or unexpected nulls early.
• Suggest validation rules automatically based on sample data, then persist and track constraint results over time to monitor data quality trends.
• Identify and quarantine rows that fail specific quality checks (e.g., invalid email format, out-of-range values) for manual review or remediation.
• Build data quality dashboards by storing verification results in a metrics repository and querying historical runs by tag and timestamp.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

PyDeequ is a Python API for Apache Spark-based data quality measurement, providing metrics computation, constraint suggestion, constraint verification, and metrics repository persistence for large datasets.

Yes. PyDeequ is actively maintained, has no known vulnerabilities, installs with low friction, and solves a concrete problem for data engineers. It is well-suited if you need declarative, scalable data quality checks. Caveat: you must have a Spark environment set up separately; PyDeequ is a wrapper, not a standalone tool.

Install

pydeequ on PyPI

pip install pydeequ

uv add pydeequ

poetry add pydeequ

Installing pydeequ

Before you install

Low install friction with a pure-Python wheel. Active maintenance with a recent release 37 days ago and last commit on 2026-07-21. Depends only on numpy and pandas at runtime.

License in practice

Licensed under Apache-2.0 (permissive), allowing commercial use, modification, and distribution with minimal restrictions beyond attribution and liability disclaimers.

Quickstart

pip install pydeequ

import pydeequ
from pydeequ.analyzers import AnalysisRunner, Size

result = AnalysisRunner(spark).onData(df).addAnalyzer(Size()).run()

Requires a Spark environment and SparkSession configured with the Deequ Maven coordinate; Spark is not installed as a runtime dependency of PyDeequ.

Verify before relying

• Whether Spark is automatically installed or must be separately configured for PyDeequ to function
• Performance characteristics and scalability limits for typical dataset sizes
• Compatibility with specific Spark versions beyond what the classifiers indicate

Package facts

Evidence: pydeequ-1.6.0-py3-none-any.whl

Keywords: deequ, pydeequ, data-engineering, data-quality, data-profiling, dataquality, dataunittest, data-unit-tests, data-profilers

Tags

More Quality Assurance packages