spark-expectations

What it is and what it does

Spark Expectations is a PySpark-native data quality framework that enforces data integrity by validating records against configurable rules before they reach downstream consumers. It supports three rule types: row-level checks (e.g., column constraints), aggregate checks (e.g., sum or average thresholds), and query-based checks (e.g., custom SQL validation). Records that fail any rule are automatically quarantined to a separate error table with metadata about which rules failed, job context, and detailed statistics, while only passing records move downstream. This prevents bad data from propagating and eliminates the need for manual error detection or separate corrective processes.

The framework integrates observability features that generate reports from statistics tables and can send alert notifications via email or Slack using customizable Jinja templates. Configuration is centralized through a Constants-based user config dictionary, making it straightforward to enable notifications, set error thresholds, and control which detailed metrics are captured. It targets Python 3.9–3.13 and depends on pluggy, pyyaml, requests, and sqlglot for plugin support, configuration parsing, HTTP calls, and SQL parsing.

Use it for:

• Enforce data contracts in ETL pipelines by rejecting rows that violate business rules and routing them to an error table for analysis.
• Monitor aggregate data quality (e.g., row counts, sums, averages) and trigger alerts when metrics fall outside acceptable ranges.
• Prevent downstream teams from consuming malformed data by automatically filtering and quarantining failed records at the source.
• Generate observability reports and email/Slack notifications summarizing data quality metrics and rule failures for each job run.
• Implement corrective workflows by isolating error records with full metadata, enabling teams to identify root causes and reprocess data.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Spark Expectations is a data quality framework that validates PySpark DataFrames against row-level, aggregate, and query-based rules, quarantining failed records to an error table while passing clean data downstream.

Yes, if you run PySpark pipelines and need built-in data quality enforcement with automatic error quarantine and observability. The low install friction, active maintenance, and absence of known vulnerabilities make it safe to adopt. However, verify the license terms first—the metadata does not declare a license identifier—and confirm that PySpark is already in your environment, as it is not listed as an explicit runtime dependency.

Install

spark-expectations on PyPI

pip install spark-expectations

uv add spark-expectations

poetry add spark-expectations

Installing spark-expectations

Before you install

Low install friction with a pure Python wheel and four lightweight runtime dependencies (pluggy, pyyaml, requests, sqlglot). Maintenance is active with a recent release within 48 days.

License in practice

License treatment is unclear—no SPDX identifier or raw license text is available in the package metadata. Verify the actual license before use in proprietary or restricted contexts.

Quickstart

pip install spark-expectations

from spark_expectations.config.user_config import Constants as user_config

se_user_conf = {
    user_config.se_notifications_enable_email: False,
    user_config.se_enable_obs_dq_report_result: True,
}

Requires Python 3.9 to 3.13 and a working PySpark environment; configuration via Constants class is mandatory before use.

Verify before relying

• Whether the package requires PySpark as a system dependency (not listed in runtime deps but core to its function).
• Actual license identifier and any redistribution or commercial-use restrictions.
• Whether email and Slack notifications require additional setup beyond the configuration shown.

Package facts

Evidence: spark_expectations-2.10.1-py3-none-any.whl

Tags

More Quality Assurance packages