pyspark-extension

What it is and what it does

pyspark-extension is a Python library that wraps and exposes Scala-based extensions to Apache Spark, providing utilities for common data transformation and inspection tasks. It includes a diff operation to compute row-level changes between datasets, histogram generation, global row numbering without window specifications, and tools to inspect Parquet file metadata. The package also enables dynamic installation of Python dependencies into running PySpark jobs via pip or Poetry, and provides helper functions for .NET DateTime conversion, null counting, and Spark job description management.

The library is primarily used during development via pip for IDE support and testing, but cluster execution requires deploying the underlying Scala JAR through Spark's dependency mechanism (spark.jars.packages, spark-submit, or notebook configuration). It has minimal runtime dependencies and supports Python 3.7 through 3.13, making it compatible with a wide range of PySpark environments.

Use it for:

• Compute row-level differences (adds, deletes, changes) between two large datasets to identify what changed.
• Generate histogram DataFrames for exploratory data analysis and distribution visualization.
• Assign global row numbers across an entire dataset without window specification overhead.
• Inspect Parquet file structure and metadata to diagnose partitioning or schema issues.
• Dynamically install Python packages into a running PySpark job without restarting the cluster.
• Convert .NET DateTime.Ticks to Spark timestamps for interoperability with C# or F# data sources.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides Python bindings and utilities for Apache Spark, including dataset diffing, histograms, global row numbering, Parquet inspection, and dynamic Python package installation into running PySpark jobs.

Yes. Active maintenance, permissive Apache License 2.0, low install friction, no known vulnerabilities, and production-stable status make it a safe choice. Install it for development and IDE support; cluster use requires the Scala JAR via spark-submit or notebook configuration. Useful if you need dataset diffing, Parquet inspection, or dynamic package installation in PySpark.

Install

pyspark-extension on PyPI

pip install pyspark-extension

uv add pyspark-extension

poetry add pyspark-extension

Installing pyspark-extension

Before you install

Low friction; single lightweight runtime dependency (typing_extensions). Active maintenance with recent commits and stable production status. Supports Python 3.7 through 3.13.

License in practice

Apache License 2.0 (permissive). Allows commercial use, modification, and distribution with minimal restrictions; suitable for proprietary projects.

Quickstart

pip install pyspark-extension==2.15.0.4.1

from pyspark.sql import SparkSession

spark = SparkSession.builder.config("spark.jars.packages", "uk.co.gresearch.spark:spark-extension_2.12:2.15.0-3.4").getOrCreate()

Requires PySpark ≥3.1.0 for runtime package installation; local development use only via pip—cluster deployment requires adding the Scala JAR via spark.jars.packages or spark-submit.

Verify before relying

• Performance characteristics when diffing or histogramming very large datasets.
• Compatibility matrix between specific pyspark-extension and PySpark minor versions.
• Whether PyPy implementation is fully supported despite CPython focus in documentation.

Package facts

Evidence: pyspark_extension-2.15.0.4.1-py3-none-any.whl

Tags

More Distributed Computing packages