pytest-spark
pytest plugin to run the tests with support of pyspark.
Decision gist · record as of 2026-08-14
Yes, if you are testing PySpark code and want a lightweight, low-friction way to inject Spark fixtures into pytest. The plugin is stable (MIT license, no known vulnerabilities) and has been in active use since its first release in 2016-12-29. However, note that maintenance is aging—the last release was 450 days ago—so verify compatibility with your specific PySpark version before relying on it for new projects.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires either pyspark installed via pip or SPARK_HOME environment variable / pytest.ini configuration to locate a local Spark installation.
- Low install friction with only two runtime dependencies (pytest and findspark).
- Maintenance status is aging—last release was 450 days ago, though the repository remains active with a recent commit on 2025-05-21 and 88 stars.
License · maintenance · safety
MIT (permissive) — MIT license (permissive) places no restrictions on use, modification, or distribution in your own projects.
last release 2025-05-21 (450 days) · last repo commit 2025-05-21 · 88 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 390,724 downloads/mo, #7,020 on PyPI
Alternatives
Verify before relying
pip install pytest-spark
# In your test file:
def test_spark_session_dataframe(spark_session):
test_df = spark_session.createDataFrame([[1,3],[2,4]], "a: int, b: int")
# test your dataframe logic- Whether the package works reliably with modern PySpark versions given classifiers list only Python 3.2–3.6.
- Current compatibility with Spark 3.4+ and Spark Connect functionality beyond what the description states.
- Actual download volume and user base size to confirm production readiness.
What it is and what it does
pytest-spark is a pytest plugin that bridges Apache Spark and PySpark into your test environment. It lets you configure Spark's location and options (like external JAR packages) via pytest.ini or command-line flags, then provides two session-scoped fixtures—spark_context and spark_session—that your tests can use to create RDDs, DataFrames, and run Spark operations. The plugin handles initialization and teardown, so you write tests as if Spark were a normal pytest fixture.
The plugin is useful when you have PySpark code to test and want to avoid boilerplate Spark setup in each test. It supports both traditional SparkContext and modern SparkSession (Spark 2.0+) with Hive support enabled by default, and also works with Spark Connect for remote execution. Configuration is straightforward: either install pyspark via pip (no SPARK_HOME needed) or point the plugin to a local Spark installation.
Use it for
- Test PySpark DataFrame transformations and SQL queries using the spark_session fixture in a standard pytest workflow.
- Run RDD-based tests with spark_context fixture for legacy or lower-level Spark code.
- Load external Spark packages (e.g., spark-xml) via spark_options in pytest.ini for integration tests.
- Test Spark code against a remote Spark Connect server by setting SPARK_REMOTE or spark_connect_url.
- Customize Spark executor and parallelism settings per test suite via pytest.ini without modifying test code.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you are testing PySpark code and want a lightweight, low-friction way to inject Spark fixtures into pytest.
The plugin is stable (MIT license, no known vulnerabilities) and has been in active use since its first release in 2016-12-29. However, note that maintenance is aging—the last release was 450 days ago—so verify compatibility with your specific PySpark version before relying on it for new projects.
Install
pytest-spark on PyPI
Before you install
Low install friction with only two runtime dependencies (pytest and findspark). Maintenance status is aging—last release was 450 days ago, though the repository remains active with a recent commit on 2025-05-21 and 88 stars.
Requires either pyspark installed via pip or SPARK_HOME environment variable / pytest.ini configuration to locate a local Spark installation.
License in practice
MIT license (permissive) places no restrictions on use, modification, or distribution in your own projects.
Quickstart
pip install pytest-spark
# In your test file:
def test_spark_session_dataframe(spark_session):
test_df = spark_session.createDataFrame([[1,3],[2,4]], "a: int, b: int")
# test your dataframe logicVerify before relying
- Whether the package works reliably with modern PySpark versions given classifiers list only Python 3.2–3.6.
- Current compatibility with Spark 3.4+ and Spark Connect functionality beyond what the description states.
- Actual download volume and user base size to confirm production readiness.
Package facts
| License | MIT permissive |
| Python support | Not specified |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 2 packagespytestfindspark |
| Maintenance | Aging 450 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 390,724 / month, #7,020 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 4 - BetaFramework :: PytestIntended Audience :: DevelopersLicense :: OSI Approved :: MIT LicenseProgramming Language :: Python :: 2Programming Language :: Python :: 2.7Programming Language :: Python :: 3Programming Language :: Python :: 3.2Programming Language :: Python :: 3.3Programming Language :: Python :: 3.4Programming Language :: Python :: 3.5Programming Language :: Python :: 3.6Topic :: Software Development :: Libraries :: Python ModulesTopic :: Software Development :: Testing |
Evidence: pytest_spark-0.8.0-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “pytest spark plugin”
- pytest-sparkA pytest plugin that integrates Apache Spark and PySpark into your…
- chispaProvides assertion methods for testing PySpark code with descriptive…
- kedro-datasetsKedro-Datasets provides data connectors for Kedro's DataCatalog,…
Give your agent the search over MCP, or paste the wish link into any chat.
More Python Modules packages
Converts domain names between Unicode and ASCII-compatible encoding (Punycode) according to IDNA 2008 and Unicode Technical Standard 46, with security validation and broader script coverage than the standard library.
Install it if you work with internationalized domain names, need to validate domains, or use HTTP clients that depend on it transitively.
Setuptools is a Python build backend and package management tool that handles building, distributing, and installing Python packages, including support for C/C++ extension modules.
PyYAML parses and emits YAML 1.1 data format, enabling serialization and deserialization of configuration files and Python objects to and from human-readable YAML text.
Pydantic validates Python data structures against type hints, coercing and checking input at runtime to ensure it matches a declared schema.
Provides reusable metadata objects for use with PEP-593 `typing.Annotated` to express common constraints like bounds, collection sizes, and predicates on types.
Install it if you use or build libraries that need to express type constraints in a standardized, inspectable way—or if you want to annotate your own types with…
Provides runtime tools to inspect and introspect Python type annotations, enabling programmatic examination of type hints at execution time.
See also findspark · pytest-rabbitmq · pyspark-test · pytest-redis · pytest-mysql · dataproc-spark-connect · pyspark-extension · pytest-postgresql · pytest-django · sagemaker-feature-store-pyspark