databricks-labs-dqx

What it is and what it does

DQX is a data quality framework for PySpark workloads on Databricks that lets you define, run, and monitor quality checks on both batch and streaming DataFrames. It supports built-in checks (null, range, regex, referential, aggregate, geo, PII) that you can apply row-level or column/dataset-level, plus custom check functions. You can define checks as code or declaratively in YAML/JSON, mark failures as warnings or errors, and route invalid data to quarantine, drop, or mark operations. The library includes data profiling, automatic rule generation from existing data, ML-based row anomaly detection, and integration with Databricks data contracts for schema validation.

Results are persisted to Delta tables with built-in aggregate metrics and a Lakeview dashboard for tracking quality over time. You can trigger Slack, Teams, or webhook alerts when metrics cross thresholds, and optionally fail pipelines on quality violations. DQX Studio provides a browser-based no-code UI for authoring and monitoring rules as a Databricks App. The framework applies the same API to batch DataFrames and Spark Structured Streaming (including DLT pipelines).

Use it for:

• Validate incoming data against business rules before loading into a data lake.
• Monitor data quality metrics on streaming pipelines and alert teams when error rates exceed thresholds.
• Generate quality rules automatically from existing datasets and refine them with LLM suggestions.
• Enforce schema and referential integrity checks as part of a Databricks data contract.
• Detect anomalous rows in large datasets and get explanations for investigation.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

DQX provides rule-based data quality checking for PySpark DataFrames on Databricks, supporting batch and streaming workloads with built-in checks, custom rules, and automated quality monitoring.

Yes, with conditions. DQX is actively maintained, has low install friction, and offers a comprehensive rule-based quality framework tailored to Databricks workloads. However, the license treatment is unclear—verify the actual license before production use. Also confirm that advanced features fit your Databricks setup and cost model. Not formally supported by Databricks with SLAs; issues are reviewed as time permits.

Install

databricks-labs-dqx on PyPI

pip install databricks-labs-dqx

uv add databricks-labs-dqx

poetry add databricks-labs-dqx

Installing databricks-labs-dqx

Before you install

Low friction: pure Python wheel with five runtime dependencies (databricks-sdk, pydantic, pyyaml, sqlalchemy, databricks-labs-blueprint). Active maintenance with a release 1 day old and recent commits. Requires Python 3.10+.

License in practice

License treatment is unclear—no SPDX identifier or raw license text is available. Verify the actual license terms before adopting in production, especially for commercial use.

Quickstart

pip install databricks-labs-dqx

from databricks_labs_dqx import DQX

dqx = DQX()
results = dqx.run_checks(df, checks=[...])

Requires Python 3.10 or later; databricks-sdk must be configured with valid Databricks workspace credentials.

Verify before relying

• Exact scope and maturity level of the built-in checks across different categories (null, range, regex, referential, aggregate, geo, PII).
• Performance characteristics and scalability limits for very large datasets or high-frequency streaming workloads.
• Whether LLM-driven rule generation and anomaly detection require additional Databricks services or incur extra costs.
• Compatibility and integration details with Databricks Unity Catalog, Volumes, and Lakebase (PostgreSQL) backends.

Package facts

Evidence: databricks_labs_dqx-0.16.0-py3-none-any.whl

Keywords: Databricks

Tags

More Libraries packages