{"categories":[{"label":"Python Modules","url":"https://skillfed.io/packages/category/software-development-libraries-python-modules/12"},{"label":"Testing","url":"https://skillfed.io/packages/category/software-development-testing/3"}],"enrichment":{"capability":"A pytest plugin that integrates Apache Spark and PySpark into your test suite, providing session-scoped fixtures for SparkContext and SparkSession with configurable Spark options.","skillfed_tags":["spark-testing","pytest-plugin","data-engineering"],"use_cases":["Test PySpark DataFrame transformations and SQL queries using the spark_session fixture in a standard pytest workflow.","Run RDD-based tests with spark_context fixture for legacy or lower-level Spark code.","Load external Spark packages (e.g., spark-xml) via spark_options in pytest.ini for integration tests.","Test Spark code against a remote Spark Connect server by setting SPARK_REMOTE or spark_connect_url.","Customize Spark executor and parallelism settings per test suite via pytest.ini without modifying test code."],"what_it_does":"pytest-spark is a pytest plugin that bridges Apache Spark and PySpark into your test environment. It lets you configure Spark's location and options (like external JAR packages) via pytest.ini or command-line flags, then provides two session-scoped fixtures\u2014spark_context and spark_session\u2014that your tests can use to create RDDs, DataFrames, and run Spark operations. The plugin handles initialization and teardown, so you write tests as if Spark were a normal pytest fixture.\n\nThe plugin is useful when you have PySpark code to test and want to avoid boilerplate Spark setup in each test. It supports both traditional SparkContext and modern SparkSession (Spark 2.0+) with Hive support enabled by default, and also works with Spark Connect for remote execution. Configuration is straightforward: either install pyspark via pip (no SPARK_HOME needed) or point the plugin to a local Spark installation.","worth_installing":"Yes, if you are testing PySpark code and want a lightweight, low-friction way to inject Spark fixtures into pytest. The plugin is stable (MIT license, no known vulnerabilities) and has been in active use since its first release in 2016-12-29. However, note that maintenance is aging\u2014the last release was 450 days ago\u2014so verify compatibility with your specific PySpark version before relying on it for new projects."},"id":"pytest-spark","links":{"html":"https://skillfed.io/packages/pytest-spark","md":"https://skillfed.io/packages/pytest-spark.md","pypi":"https://pypi.org/project/pytest-spark/"},"maintenance":{"status":"aging"},"meta":{"latest_release":"2025-05-21","license_spdx":null,"license_treatment":"permissive","name":"pytest-spark","python_support":"unspecified","summary":"pytest plugin to run the tests with support of pyspark."},"popularity":{"monthly_downloads":390724,"position":7020,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.8.0"}