pyspark-pandas

What it is and what it does

pyspark-pandas aims to bridge Pandas and Apache Spark by providing utilities to distribute Pandas DataFrames and Series across Spark clusters. It was designed to enable data analysis workflows that combine Pandas' ease-of-use with Spark's distributed processing power for handling large datasets.

However, this project has been abandoned since 2014-10-14 and is no longer maintained. The package itself explicitly directs users to consider alternatives instead, suggesting this codebase may have been superseded. With no runtime dependencies listed and no recent activity, it represents a snapshot from the early Spark-Pandas integration era.

Use it for:

• Distributing Pandas DataFrames across Spark clusters for parallel processing of large datasets
• Combining Pandas data manipulation with Spark's distributed computing for legacy workflows
• Prototyping distributed data analysis before migrating to modern solutions

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides tools for distributing Pandas DataFrames and Series across Apache Spark clusters for large-scale data processing.

No. This package is abandoned (last release 2014-10-14, no commits since), has an unclear license, and explicitly recommends users evaluate alternatives instead. Modern alternatives are far better choices for any current use case.

Install

pyspark-pandas on PyPI

pip install pyspark-pandas

uv add pyspark-pandas

poetry add pyspark-pandas

Installing pyspark-pandas

Before you install

High install friction and abandoned maintenance status. Last release was 2014-10-14, with no commits since then. The project itself recommends evaluating alternatives before adopting this package.

License in practice

License is marked UNKNOWN with no SPDX identifier, creating unclear legal standing for use or redistribution.

Quickstart

pip install pyspark-pandas==0.0.7
import pyspark_pandas

No runtime dependencies declared; compatibility with modern PySpark and Pandas versions is unknown given the 2014-10-14 release date.

Verify before relying

• Whether this package is compatible with modern versions of PySpark and Pandas
• What Python versions this package actually supports (unspecified in metadata)
• Whether the codebase has any undisclosed security issues despite zero OSV records

Package facts

Evidence: pyspark-pandas-0.0.7.zip

Keywords: spark, pyspark, pandas, dataframe, series

Tags

More Distributed Computing packages