sparkaid

What it is and what it does

Sparkaid is a utility library for PySpark that simplifies working with DataFrames containing complex nested schemas. It provides functions to flatten StructType columns (removing nesting layers), rename fields within nested structures, and convert JSON schemas to Spark StructType objects. The package addresses common pain points when working with nested data: complex SQL queries, difficulty renaming or casting nested columns, and unnecessary I/O overhead when reading only specific nested columns from Parquet files.

The library's core feature is its flatten() function, which unpacks nested StructType columns into flat columns with configurable separators (e.g., converting {"parent": {"child": "value"}} into {"parent_child": "value"}). Version 1.0.0 introduced a breaking change where flatten() now stops at ArrayType columns by default, requiring explicit configuration to unpack arrays. The package depends solely on pyspark and installs as a pure Python wheel with minimal friction.

Use it for:

• Flatten deeply nested JSON or Parquet data into a single-level schema for simpler SQL queries and easier column access.
• Rename nested struct fields programmatically without manually reconstructing the entire schema using struct() or cast().
• Convert JSON schema definitions to Spark StructType objects for schema validation or DataFrame creation.
• Reduce I/O overhead when reading only specific nested columns from Parquet files by flattening the schema first.
• Transform snake_case naming conventions across nested column hierarchies in a single operation.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides utilities for working with Spark DataFrames that have complex nested schemas, including flattening StructType columns, renaming nested fields, and converting JSON schemas to Spark schemas.

No. The package is abandoned (last release August 2022, no commits since September 2022) and carries a copyleft license that may restrict use in proprietary projects. While it addresses a real problem in Spark schema manipulation, modern Spark versions (3.x+) and the Databricks documentation now provide native or better-maintained alternatives. Install only if you are locked into an older Spark version and cannot upgrade.

Install

sparkaid on PyPI

pip install sparkaid

uv add sparkaid

poetry add sparkaid

Installing sparkaid

Before you install

Low install friction with a single runtime dependency on pyspark. However, the package is abandoned—last release was 2022-08-29 and last commit 2022-09-16—so no maintenance or bug fixes should be expected.

License in practice

Licensed under LGPLv3+, a copyleft license requiring derivative works to be distributed under the same terms. This may restrict use in proprietary or closed-source projects.

Quickstart

pip install sparkaid

from sparkaid import flatten
from pyspark.sql import Row, SparkSession

spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([Row(structA=Row(field1=10, field2=1.5))])
flattened = flatten(df)

Requires pyspark to be installed and a working Spark environment; package is abandoned and may not be compatible with recent Spark versions.

Verify before relying

• Compatibility with Spark versions released after 2022-09-16 is unknown.
• Whether the snake_case() and json_schema_to_spark_schema() functions mentioned in the changelog are fully documented and stable.
• Performance characteristics when flattening very large or deeply nested schemas.

Package facts

Evidence: sparkaid-1.0.0-py3-none-any.whl

Keywords: SPARK, DATAFRAME, MANIPULATE

Tags

More Build Tools packages