{"categories":[{"label":"Testing","url":"https://skillfed.io/packages/category/software-development-testing/3"}],"enrichment":{"capability":"Provides a testing utility to assert equality between two PySpark DataFrames with configurable strictness for data types, column names, and column ordering.","skillfed_tags":["spark-testing","dataframe-comparison"],"use_cases":["Unit testing Spark ETL jobs by asserting that transformed DataFrames match expected output.","Validating data pipeline stages by comparing intermediate DataFrames with known-good reference data.","Regression testing after schema changes by checking both data and column metadata.","Comparing DataFrames with flexible strictness\u2014e.g., ignoring column order in some tests but enforcing it in others."],"what_it_does":"pyspark-test is a lightweight assertion library for comparing PySpark DataFrames in unit tests. It wraps the core comparison logic into a single function, `assert_pyspark_df_equal`, inspired by pandas' testing module but adapted for Spark's distributed DataFrame API. The function checks row-by-row equality and can optionally verify data types, column names, and column ordering; it also supports sorting DataFrames before comparison via an `order_by` parameter.\n\nThe package is intended as a drop-in testing helper for Spark-based data pipelines. It depends only on pyspark and installs as a pure Python wheel with no compiled dependencies. However, the project is dormant\u2014no releases since October 2021 and no commits since December 2023\u2014so it may not work with recent PySpark versions or handle edge cases that have emerged since then.","worth_installing":"Yes, if you are writing unit tests for PySpark pipelines and need a simple DataFrame comparison tool. The low install friction and permissive license make it easy to adopt. However, be aware that the project is dormant and may not support the latest PySpark versions\u2014verify compatibility with your environment before relying on it in production test suites."},"id":"pyspark-test","links":{"html":"https://skillfed.io/packages/pyspark-test","md":"https://skillfed.io/packages/pyspark-test.md","pypi":"https://pypi.org/project/pyspark-test/"},"maintenance":{"status":"dormant"},"meta":{"latest_release":"2021-10-31","license_spdx":null,"license_treatment":"permissive","name":"pyspark-test","python_support":"unspecified","summary":"Check that left and right spark DataFrame are equal."},"popularity":{"monthly_downloads":377616,"position":7122,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.2.0"}