pyspark

Apache Spark Python API

pyspark Permissive license Apache-2.0 Active ★ 43,867 v4.2.0 released 2026-07-14

Install

pyspark on PyPI

pip

pip install pyspark

uv

uv add pyspark

poetry

poetry add pyspark

Package facts

License	Apache-2.0 (permissive)
Python support	supports the current Python release (>=3.10)
Install friction	high — source build required
Runtime dependencies	1 — py4j
Maintenance	actively maintained — 30 days since the last release
Last repo commit	2026-08-13
First released	2017-07-12
Popularity	one of the top 1,000 most-downloaded packages on PyPI (30-day window, as of 2026-08-13)
Known vulnerabilities	none known (OSV.dev, checked 2026-08-13)

Evidence: pyspark-4.2.0.tar.gz

Development Status :: 5 - Production/StableProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Programming Language :: Python :: Implementation :: CPythonTyping :: Typed

About pyspark

from the package's own PyPI description — quoted content, verbatim

Apache Spark

Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.

<https://spark.apache.org/>

Online Documentation

You can find the latest Spark documentation, including a programming guide, on the project web page

Python Packaging

This README file only contains basic information related to pip installed PySpark. This packaging is currently experimental and may change in future versions (although we will do our best to keep compatibility). Using PySpark requires the Spark JARs, and if you are building this from source please see the builder instructions at "Building Spark".

The Python packaging for Spark is not intended to replace all of the other use cases. This...

Read as markdown · JSON record · Source repository · Homepage

AI interpretation — verify before relying

AI-generated interpretation of the package facts above; every digit, version, license, or vulnerability id it cites is grounded in the facts already shown on this page

PySpark is Apache Spark's Python API for distributed data processing and analytics at scale, supporting SQL queries, machine learning, graph processing, and stream processing through high-level APIs.

High install friction due to large package size and Java/Spark JAR dependencies. Actively maintained with recent releases (30 days since last update), supporting Python 3.10–3.14.

Apache-2.0 permissive license allows broad commercial and private use with minimal restrictions, though derivative works must include license and copyright notices.

Usage

pip install pyspark==4.2.0
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('example').getOrCreate()
df = spark.createDataFrame([(1, 'a'), (2, 'b')], ['id', 'letter'])
df.show()

Requires Java Runtime Environment (JRE) and Spark JARs; this pip package is experimental and intended for cluster interaction, not standalone cluster setup.

Verdict: PySpark 4.2.0 is production-stable and actively maintained with no known vulnerabilities, but carries substantial install friction from its Java/Spark dependencies. Suitable for developers working with existing Spark clusters or large-scale distributed analytics, though the experimental nature of the pip packaging and Java requirement warrant careful integration planning.

Needs verification

Community adoption and ecosystem maturity beyond the fact sheet's maintenance signals
Whether the py4j dependency version is pinned and compatible across Spark cluster versions
Performance characteristics and memory overhead of the pip-installed variant versus full Spark distribution

distributed data processing pythonspark python apilarge scale analytics enginedistributed machine learningspark sql dataframesstream processing pythondistributed graph processing