pyspark-test

What it is and what it does

pyspark-test is a lightweight assertion library for comparing PySpark DataFrames in unit tests. It wraps the core comparison logic into a single function, `assert_pyspark_df_equal`, inspired by pandas' testing module but adapted for Spark's distributed DataFrame API. The function checks row-by-row equality and can optionally verify data types, column names, and column ordering; it also supports sorting DataFrames before comparison via an `order_by` parameter.

The package is intended as a drop-in testing helper for Spark-based data pipelines. It depends only on pyspark and installs as a pure Python wheel with no compiled dependencies. However, the project is dormant—no releases since October 2021 and no commits since December 2023—so it may not work with recent PySpark versions or handle edge cases that have emerged since then.

Use it for:

• Unit testing Spark ETL jobs by asserting that transformed DataFrames match expected output.
• Validating data pipeline stages by comparing intermediate DataFrames with known-good reference data.
• Regression testing after schema changes by checking both data and column metadata.
• Comparing DataFrames with flexible strictness—e.g., ignoring column order in some tests but enforcing it in others.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides a testing utility to assert equality between two PySpark DataFrames with configurable strictness for data types, column names, and column ordering.

Yes, if you are writing unit tests for PySpark pipelines and need a simple DataFrame comparison tool. The low install friction and permissive license make it easy to adopt. However, be aware that the project is dormant and may not support the latest PySpark versions—verify compatibility with your environment before relying on it in production test suites.

Install

pyspark-test on PyPI

pip install pyspark-test

uv add pyspark-test

poetry add pyspark-test

Installing pyspark-test

Before you install

Low install friction with a single runtime dependency on pyspark. Maintenance is dormant—last release was in 2021 and last commit in December 2023—so expect no active bug fixes or updates.

License in practice

Licensed under Apache 2.0 (permissive), so you may use, modify, and distribute freely with minimal restrictions.

Quickstart

from pyspark_test import assert_pyspark_df_equal

assert_pyspark_df_equal(left_df, right_df, check_dtype=True, check_column_names=False)

Requires pyspark to be installed and a Spark context to be available at runtime.

Verify before relying

• Whether the package handles edge cases like very large DataFrames or complex nested schemas.
• Current compatibility with recent PySpark versions (last release was 2021).
• Performance characteristics when comparing DataFrames with millions of rows.

Package facts

Evidence: pyspark_test-0.2.0-py3-none-any.whl

Keywords: assert, pyspark, unit, test, testing, compare

Tags

More Testing packages