pyspark-stubs

What it is and what it does

PySpark Stubs is a collection of type stub files (.pyi) for Apache Spark's Python API, generated by mypy's stubgen tool and manually refined with accurate type hints. It allows developers to use static type checkers like mypy and IDEs like PyCharm to catch type errors in PySpark code before runtime, and enables better autocompletion in editors.

The package was originally maintained as a standalone project but has been merged into the main Apache Spark repository for versions 3.1 and later. This PyPI package continues to serve Spark 2.4 and 3.0 users, though it is no longer actively developed. Installation overlays the stub files into your PySpark installation directory so type checkers can find them automatically.

Use it for:

• Catch type errors in PySpark code during development using mypy or similar type checkers before submitting to production.
• Improve IDE autocompletion and inline documentation in PyCharm, VS Code, or other editors that support type hints.
• Enable static analysis in CI/CD pipelines to enforce type safety on Spark data processing jobs.
• Support legacy Spark 2.4 or 3.0 projects that need type hints but cannot upgrade to newer Spark versions with built-in stubs.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides type stubs for PySpark to enable static type checking and improved IDE autocompletion when working with Apache Spark in Python.

Yes, if you are using Spark 2.4 or 3.0 and want type checking support—but be aware the package is abandoned and no longer receives updates. For Spark 3.1+, use the type hints built into Apache Spark itself instead. The stubs are permissively licensed and install easily, but verify compatibility with your current mypy and PySpark versions before relying on them in production.

Install

pyspark-stubs on PyPI

pip install pyspark-stubs

uv add pyspark-stubs

poetry add pyspark-stubs

Installing pyspark-stubs

Before you install

Installation is straightforward with low friction, but the package is abandoned as of August 2022 and no longer maintained. Development was merged into Apache Spark 3.1+; this package only receives fixes for Spark 2.4 and 3.0 until their deprecation.

License in practice

Licensed under the permissive Apache License, which allows use in commercial and private projects with minimal restrictions.

Quickstart

pip install pyspark-stubs

# Type checker (e.g., mypy) will then recognize PySpark types
import pyspark
from pyspark.sql import SparkSession

Requires a type checker (mypy, pytype, or equivalent) to be installed separately to use the stubs; stubs alone do not enforce types at runtime.

Verify before relying

• Whether stub coverage remains adequate for modern PySpark workflows given the package is unmaintained since 2022.
• Compatibility with PySpark versions released after January 2022 (the latest release date).
• Whether the stubs work correctly with current versions of mypy or other type checkers, as the description notes rare incompatibilities with preceding mypy releases.

Package facts

Evidence: pyspark_stubs-3.0.0.post3-py3-none-any.whl

Tags

More Quality Assurance packages