---
id: pyspark-extension
version: "2.15.0.4.1"
license: http://www.apache.org/licenses/LICENSE-2.0.html
license_treatment: permissive
maintenance: active
---

# pyspark-extension — A library that provides useful extensions to Apache Spark.

License: permissive · Maintenance: active · Downloads: 92.9K/mo

## What it is and what it does

pyspark-extension is a Python library that wraps and exposes Scala-based extensions to Apache Spark, providing utilities for common data transformation and inspection tasks. It includes a diff operation to compute row-level changes between datasets, histogram generation, global row numbering without window specifications, and tools to inspect Parquet file metadata. The package also enables dynamic installation of Python dependencies into running PySpark jobs via pip or Poetry, and provides helper functions for .NET DateTime conversion, null counting, and Spark job description management.

The library is primarily used during development via pip for IDE support and testing, but cluster execution requires deploying the underlying Scala JAR through Spark's dependency mechanism (spark.jars.packages, spark-submit, or notebook configuration). It has minimal runtime dependencies and supports Python 3.7 through 3.13, making it compatible with a wide range of PySpark environments.

Use it for:

- Compute row-level differences (adds, deletes, changes) between two large datasets to identify what changed.
- Generate histogram DataFrames for exploratory data analysis and distribution visualization.
- Assign global row numbers across an entire dataset without window specification overhead.
- Inspect Parquet file structure and metadata to diagnose partitioning or schema issues.
- Dynamically install Python packages into a running PySpark job without restarting the cluster.
- Convert .NET DateTime.Ticks to Spark timestamps for interoperability with C# or F# data sources.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Provides Python bindings and utilities for Apache Spark, including dataset diffing, histograms, global row numbering, Parquet inspection, and dynamic Python package installation into running PySpark jobs.

Yes. Active maintenance, permissive Apache License 2.0, low install friction, no known vulnerabilities, and production-stable status make it a safe choice. Install it for development and IDE support; cluster use requires the Scala JAR via spark-submit or notebook configuration. Useful if you need dataset diffing, Parquet inspection, or dynamic package installation in PySpark.

## Install

pip install pyspark-extension
uv add pyspark-extension
poetry add pyspark-extension

## Installing pyspark-extension

Before you install: Low friction; single lightweight runtime dependency (typing_extensions). Active maintenance with recent commits and stable production status. Supports Python 3.7 through 3.13.

License in practice: Apache License 2.0 (permissive). Allows commercial use, modification, and distribution with minimal restrictions; suitable for proprietary projects.

Quickstart:

pip install pyspark-extension==2.15.0.4.1

from pyspark.sql import SparkSession

spark = SparkSession.builder.config("spark.jars.packages", "uk.co.gresearch.spark:spark-extension_2.12:2.15.0-3.4").getOrCreate()

Requires PySpark ≥3.1.0 for runtime package installation; local development use only via pip—cluster deployment requires adding the Scala JAR via spark.jars.packages or spark-submit.

Verify before relying:

- Performance characteristics when diffing or histogramming very large datasets.
- Compatibility matrix between specific pyspark-extension and PySpark minor versions.
- Whether PyPy implementation is fully supported despite CPython focus in documentation.

## Package facts

- License: http://www.apache.org/licenses/LICENSE-2.0.html (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 92.9K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

spark dataset diff, pyspark extensions utilities, spark histogram transformation, global row number spark, parquet file inspection, install packages pyspark runtime, spark temporary directory, spark-extensions, data-transformation, distributed-computing

[View on SkillFed](https://skillfed.io/packages/pyspark-extension) · [View on PyPI](https://pypi.org/project/pyspark-extension/)