pytest-spark

What it is and what it does

pytest-spark is a pytest plugin that bridges Apache Spark and PySpark into your test environment. It lets you configure Spark's location and options (like external JAR packages) via pytest.ini or command-line flags, then provides two session-scoped fixtures—spark_context and spark_session—that your tests can use to create RDDs, DataFrames, and run Spark operations. The plugin handles initialization and teardown, so you write tests as if Spark were a normal pytest fixture.

The plugin is useful when you have PySpark code to test and want to avoid boilerplate Spark setup in each test. It supports both traditional SparkContext and modern SparkSession (Spark 2.0+) with Hive support enabled by default, and also works with Spark Connect for remote execution. Configuration is straightforward: either install pyspark via pip (no SPARK_HOME needed) or point the plugin to a local Spark installation.

Use it for:

• Test PySpark DataFrame transformations and SQL queries using the spark_session fixture in a standard pytest workflow.
• Run RDD-based tests with spark_context fixture for legacy or lower-level Spark code.
• Load external Spark packages (e.g., spark-xml) via spark_options in pytest.ini for integration tests.
• Test Spark code against a remote Spark Connect server by setting SPARK_REMOTE or spark_connect_url.
• Customize Spark executor and parallelism settings per test suite via pytest.ini without modifying test code.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

A pytest plugin that integrates Apache Spark and PySpark into your test suite, providing session-scoped fixtures for SparkContext and SparkSession with configurable Spark options.

Yes, if you are testing PySpark code and want a lightweight, low-friction way to inject Spark fixtures into pytest. The plugin is stable (MIT license, no known vulnerabilities) and has been in active use since its first release in 2016-12-29. However, note that maintenance is aging—the last release was 450 days ago—so verify compatibility with your specific PySpark version before relying on it for new projects.

Install

pytest-spark on PyPI

pip install pytest-spark

uv add pytest-spark

poetry add pytest-spark

Installing pytest-spark

Before you install

Low install friction with only two runtime dependencies (pytest and findspark). Maintenance status is aging—last release was 450 days ago, though the repository remains active with a recent commit on 2025-05-21 and 88 stars.

License in practice

MIT license (permissive) places no restrictions on use, modification, or distribution in your own projects.

Quickstart

pip install pytest-spark

# In your test file:
def test_spark_session_dataframe(spark_session):
    test_df = spark_session.createDataFrame([[1,3],[2,4]], "a: int, b: int")
    # test your dataframe logic

Requires either pyspark installed via pip or SPARK_HOME environment variable / pytest.ini configuration to locate a local Spark installation.

Verify before relying

• Whether the package works reliably with modern PySpark versions given classifiers list only Python 3.2–3.6.
• Current compatibility with Spark 3.4+ and Spark Connect functionality beyond what the description states.
• Actual download volume and user base size to confirm production readiness.

Package facts

Evidence: pytest_spark-0.8.0-py3-none-any.whl

Keywords: pytest, spark, pyspark, unittest, test

Tags

More Python Modules packages