pyddq

What it is and what it does

PyDDQ is a Python wrapper around the Drunken Data Quality (DDQ) Scala library, enabling data quality validation on Spark DataFrames through a fluent constraint-checking API. It lets you define and run checks for row counts, unique keys, foreign keys, nullability, and custom SQL expressions, then report results to stdout or custom reporters in formats like Markdown or console output.

The package is designed for continuous data import pipelines where you need to assert data quality before processing. However, it requires a pre-configured PySpark environment with the DDQ jar file loaded via the driver class path—it cannot be installed as a standalone Python package. The project has been abandoned since its last release on 2020-02-28, making it unsuitable for modern production environments without significant verification and potential maintenance work.

Use it for:

• Validate imported data meets row count and uniqueness constraints before ETL processing begins.
• Write automated quality tests that inspect constraint results programmatically and fail data loads on violations.
• Generate Markdown or console reports on data quality checks across multiple Spark DataFrames in a single run.
• Enforce referential integrity by checking foreign key relationships between Spark tables.
• Test that required columns are never null and custom SQL expressions hold true across the dataset.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

PyDDQ is a Python API for running data quality constraint checks on Spark DataFrames, validating row counts, uniqueness, nullability, foreign keys, and custom SQL expressions.

No. The project is abandoned (last update 2020-02-28), requires complex pre-configuration of a PySpark environment with external jar files, and has received no maintenance. Modern alternatives and actively maintained data validation libraries are strongly preferred for any production or new development work.

Install

pyddq on PyPI

pip install pyddq

uv add pyddq

poetry add pyddq

Installing pyddq

Before you install

Installation friction is high: the package requires a pre-configured PySpark environment with the DDQ jar file already available, and the project has been abandoned since its last release with no maintenance activity.

License in practice

Licensed under Apache License Version 2.0 (permissive), allowing commercial use and modification without restriction, though the abandoned status means no ongoing legal or security updates.

Quickstart

# Requires PySpark with DDQ jar pre-configured
pyspark --driver-class-path drunken-data-quality_2.11-5.0.0.jar

from pyddq.core import Check

df = spark.createDataFrame([(1, "a"), (1, None), (3, "c")])
check = Check(df)
check.hasUniqueKey("_1", "_2").isNeverNull("_1").run()

Requires PySpark environment with DDQ jar file already added to the driver class path; cannot be installed via pip alone without separate Spark and jar setup.

Verify before relying

• Whether the package works with Spark versions beyond 2.2.x (last tested version per documentation)
• Python version compatibility (listed as unspecified in metadata)
• Whether jar dependencies remain available or accessible from original sources

Package facts

Evidence: pyddq-5.0.0.tar.gz

Tags

More Quality Assurance packages