---
id: pyspark-test
version: "0.2.0"
license: Apache Software License (Apache 2.0)
license_treatment: permissive
maintenance: dormant
---

# pyspark-test — Check that left and right spark DataFrame are equal.

License: permissive · Maintenance: dormant · Downloads: 377.6K/mo

## What it is and what it does

pyspark-test is a lightweight assertion library for comparing PySpark DataFrames in unit tests. It wraps the core comparison logic into a single function, `assert_pyspark_df_equal`, inspired by pandas' testing module but adapted for Spark's distributed DataFrame API. The function checks row-by-row equality and can optionally verify data types, column names, and column ordering; it also supports sorting DataFrames before comparison via an `order_by` parameter.

The package is intended as a drop-in testing helper for Spark-based data pipelines. It depends only on pyspark and installs as a pure Python wheel with no compiled dependencies. However, the project is dormant—no releases since October 2021 and no commits since December 2023—so it may not work with recent PySpark versions or handle edge cases that have emerged since then.

Use it for:

- Unit testing Spark ETL jobs by asserting that transformed DataFrames match expected output.
- Validating data pipeline stages by comparing intermediate DataFrames with known-good reference data.
- Regression testing after schema changes by checking both data and column metadata.
- Comparing DataFrames with flexible strictness—e.g., ignoring column order in some tests but enforcing it in others.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Provides a testing utility to assert equality between two PySpark DataFrames with configurable strictness for data types, column names, and column ordering.

Yes, if you are writing unit tests for PySpark pipelines and need a simple DataFrame comparison tool. The low install friction and permissive license make it easy to adopt. However, be aware that the project is dormant and may not support the latest PySpark versions—verify compatibility with your environment before relying on it in production test suites.

## Install

pip install pyspark-test
uv add pyspark-test
poetry add pyspark-test

## Installing pyspark-test

Before you install: Low install friction with a single runtime dependency on pyspark. Maintenance is dormant—last release was in 2021 and last commit in December 2023—so expect no active bug fixes or updates.

License in practice: Licensed under Apache 2.0 (permissive), so you may use, modify, and distribute freely with minimal restrictions.

Quickstart:

from pyspark_test import assert_pyspark_df_equal

assert_pyspark_df_equal(left_df, right_df, check_dtype=True, check_column_names=False)

Requires pyspark to be installed and a Spark context to be available at runtime.

Verify before relying:

- Whether the package handles edge cases like very large DataFrames or complex nested schemas.
- Current compatibility with recent PySpark versions (last release was 2021).
- Performance characteristics when comparing DataFrames with millions of rows.

## Package facts

- License: Apache Software License (Apache 2.0) (permissive)
- Python support: unspecified
- Install friction: low
- Maintenance: dormant
- Downloads: 377.6K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

pyspark dataframe comparison testing, spark dataframe equality assertion, pyspark unit test helpers, dataframe diff checker, spark testing utilities, spark-testing, dataframe-comparison

[View on SkillFed](https://skillfed.io/packages/pyspark-test) · [View on PyPI](https://pypi.org/project/pyspark-test/)