{"categories":[{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing/3"}],"enrichment":{"capability":"Provides Python bindings and utilities for Apache Spark, including dataset diffing, histograms, global row numbering, Parquet inspection, and dynamic Python package installation into running PySpark jobs.","skillfed_tags":["spark-extensions","data-transformation","distributed-computing"],"use_cases":["Compute row-level differences (adds, deletes, changes) between two large datasets to identify what changed.","Generate histogram DataFrames for exploratory data analysis and distribution visualization.","Assign global row numbers across an entire dataset without window specification overhead.","Inspect Parquet file structure and metadata to diagnose partitioning or schema issues.","Dynamically install Python packages into a running PySpark job without restarting the cluster.","Convert .NET DateTime.Ticks to Spark timestamps for interoperability with C# or F# data sources."],"what_it_does":"pyspark-extension is a Python library that wraps and exposes Scala-based extensions to Apache Spark, providing utilities for common data transformation and inspection tasks. It includes a diff operation to compute row-level changes between datasets, histogram generation, global row numbering without window specifications, and tools to inspect Parquet file metadata. The package also enables dynamic installation of Python dependencies into running PySpark jobs via pip or Poetry, and provides helper functions for .NET DateTime conversion, null counting, and Spark job description management.\n\nThe library is primarily used during development via pip for IDE support and testing, but cluster execution requires deploying the underlying Scala JAR through Spark's dependency mechanism (spark.jars.packages, spark-submit, or notebook configuration). It has minimal runtime dependencies and supports Python 3.7 through 3.13, making it compatible with a wide range of PySpark environments.","worth_installing":"Yes. Active maintenance, permissive Apache License 2.0, low install friction, no known vulnerabilities, and production-stable status make it a safe choice. Install it for development and IDE support; cluster use requires the Scala JAR via spark-submit or notebook configuration. Useful if you need dataset diffing, Parquet inspection, or dynamic package installation in PySpark."},"id":"pyspark-extension","links":{"html":"https://skillfed.io/packages/pyspark-extension","md":"https://skillfed.io/packages/pyspark-extension.md","pypi":"https://pypi.org/project/pyspark-extension/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-03-18","license_spdx":null,"license_treatment":"permissive","name":"pyspark-extension","python_support":"supports_current","summary":"A library that provides useful extensions to Apache Spark."},"popularity":{"monthly_downloads":92910,"position":13415,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"2.15.0.4.1"}