pyspark-test
Check that left and right spark DataFrame are equal.
Decision gist · record as of 2026-08-14
Yes, if you are writing unit tests for PySpark pipelines and need a simple DataFrame comparison tool. The low install friction and permissive license make it easy to adopt. However, be aware that the project is dormant and may not support the latest PySpark versions—verify compatibility with your environment before relying on it in production test suites.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires pyspark to be installed and a Spark context to be available at runtime.
- Low install friction with a single runtime dependency on pyspark.
- Maintenance is dormant—last release was in 2021 and last commit in December 2023—so expect no active bug fixes or updates.
License · maintenance · safety
permissive license (permissive) — Licensed under Apache 2.0 (permissive), so you may use, modify, and distribute freely with minimal restrictions.
last release 2021-10-31 (1748 days) · last repo commit 2023-12-18 · 21 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 377,616 downloads/mo, #7,122 on PyPI
Alternatives
Verify before relying
from pyspark_test import assert_pyspark_df_equal
assert_pyspark_df_equal(left_df, right_df, check_dtype=True, check_column_names=False)- Whether the package handles edge cases like very large DataFrames or complex nested schemas.
- Current compatibility with recent PySpark versions (last release was 2021).
- Performance characteristics when comparing DataFrames with millions of rows.
What it is and what it does
pyspark-test is a lightweight assertion library for comparing PySpark DataFrames in unit tests. It wraps the core comparison logic into a single function, `assert_pyspark_df_equal`, inspired by pandas' testing module but adapted for Spark's distributed DataFrame API. The function checks row-by-row equality and can optionally verify data types, column names, and column ordering; it also supports sorting DataFrames before comparison via an `order_by` parameter.
The package is intended as a drop-in testing helper for Spark-based data pipelines. It depends only on pyspark and installs as a pure Python wheel with no compiled dependencies. However, the project is dormant—no releases since October 2021 and no commits since December 2023—so it may not work with recent PySpark versions or handle edge cases that have emerged since then.
Use it for
- Unit testing Spark ETL jobs by asserting that transformed DataFrames match expected output.
- Validating data pipeline stages by comparing intermediate DataFrames with known-good reference data.
- Regression testing after schema changes by checking both data and column metadata.
- Comparing DataFrames with flexible strictness—e.g., ignoring column order in some tests but enforcing it in others.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you are writing unit tests for PySpark pipelines and need a simple DataFrame comparison tool.
The low install friction and permissive license make it easy to adopt. However, be aware that the project is dormant and may not support the latest PySpark versions—verify compatibility with your environment before relying on it in production test suites.
Install
pyspark-test on PyPI
Before you install
Low install friction with a single runtime dependency on pyspark. Maintenance is dormant—last release was in 2021 and last commit in December 2023—so expect no active bug fixes or updates.
Requires pyspark to be installed and a Spark context to be available at runtime.
License in practice
Licensed under Apache 2.0 (permissive), so you may use, modify, and distribute freely with minimal restrictions.
Quickstart
from pyspark_test import assert_pyspark_df_equal
assert_pyspark_df_equal(left_df, right_df, check_dtype=True, check_column_names=False)Verify before relying
- Whether the package handles edge cases like very large DataFrames or complex nested schemas.
- Current compatibility with recent PySpark versions (last release was 2021).
- Performance characteristics when comparing DataFrames with millions of rows.
Package facts
| License | permissive license permissive |
| Python support | Not specified |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 1 packagepyspark |
| Maintenance | Dormant 1,748 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 377,616 / month, #7,122 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
Evidence: pyspark_test-0.2.0-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “pyspark dataframe comparison testing”
- pyspark-testProvides a testing utility to assert equality between two PySpark…
- chispaProvides assertion methods for testing PySpark code with descriptive…
- cualleeCuallee provides a dataframe-agnostic API to define and run data…
Give your agent the search over MCP, or paste the wish link into any chat.
More Testing packages
Pluggy provides a plugin system that lets you define hook specifications and register implementations to be called in sequence, enabling extensible Python applications without tight coupling.
Install it if you're building an extensible application or framework.
pytest is a testing framework that lets you write test functions using plain assert statements and automatically discovers and runs them, with detailed failure reporting.
virtualenv creates isolated Python environments where packages can be installed independently without affecting the system Python or other projects.
Coverage.py measures which lines of Python code are executed during test runs, reporting coverage percentages and identifying untested code paths.
Install it if you want to measure test completeness or enforce coverage thresholds in your project.
pytest-asyncio is a pytest plugin that enables writing and running async test functions using the asyncio library, allowing developers to await code directly within test cases.
Install it if you write tests for any asyncio-based code.
A pytest plugin that generates test reports in Common Test Report Format (CTRF) as JSON, compatible with pytest-xdist and pytest-playwright for distributed and browser-based testing.
Install it if you need CTRF-formatted test output for CI/CD integration or cross-tool reporting.
See also chispa · datacompy · databricks-test · pytest-spark · dirty-equals · spark-expectations · quinn · cuallee · pyspark-extension · findspark