{"categories":[{"label":"Quality Assurance","url":"https://skillfed.io/packages/category/software-development-quality-assurance/2"}],"enrichment":{"capability":"Spark Expectations is a data quality framework that validates PySpark DataFrames against row-level, aggregate, and query-based rules, quarantining failed records to an error table while passing clean data downstream.","skillfed_tags":["data-quality","pyspark","pipeline-validation"],"use_cases":["Enforce data contracts in ETL pipelines by rejecting rows that violate business rules and routing them to an error table for analysis.","Monitor aggregate data quality (e.g., row counts, sums, averages) and trigger alerts when metrics fall outside acceptable ranges.","Prevent downstream teams from consuming malformed data by automatically filtering and quarantining failed records at the source.","Generate observability reports and email/Slack notifications summarizing data quality metrics and rule failures for each job run.","Implement corrective workflows by isolating error records with full metadata, enabling teams to identify root causes and reprocess data."],"what_it_does":"Spark Expectations is a PySpark-native data quality framework that enforces data integrity by validating records against configurable rules before they reach downstream consumers. It supports three rule types: row-level checks (e.g., column constraints), aggregate checks (e.g., sum or average thresholds), and query-based checks (e.g., custom SQL validation). Records that fail any rule are automatically quarantined to a separate error table with metadata about which rules failed, job context, and detailed statistics, while only passing records move downstream. This prevents bad data from propagating and eliminates the need for manual error detection or separate corrective processes.\n\nThe framework integrates observability features that generate reports from statistics tables and can send alert notifications via email or Slack using customizable Jinja templates. Configuration is centralized through a Constants-based user config dictionary, making it straightforward to enable notifications, set error thresholds, and control which detailed metrics are captured. It targets Python 3.9\u20133.13 and depends on pluggy, pyyaml, requests, and sqlglot for plugin support, configuration parsing, HTTP calls, and SQL parsing.","worth_installing":"Yes, if you run PySpark pipelines and need built-in data quality enforcement with automatic error quarantine and observability. The low install friction, active maintenance, and absence of known vulnerabilities make it safe to adopt. However, verify the license terms first\u2014the metadata does not declare a license identifier\u2014and confirm that PySpark is already in your environment, as it is not listed as an explicit runtime dependency."},"id":"spark-expectations","links":{"html":"https://skillfed.io/packages/spark-expectations","md":"https://skillfed.io/packages/spark-expectations.md","pypi":"https://pypi.org/project/spark-expectations/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-06-27","license_spdx":null,"license_treatment":"unclear","name":"spark-expectations","python_support":"supports_current","summary":"This project helps us to run Data Quality Rules in flight while spark job is being run"},"popularity":{"monthly_downloads":489853,"position":6374,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"2.10.1"}