{"categories":[{"label":"Build Tools","url":"https://skillfed.io/packages/category/software-development-build-tools/4"}],"enrichment":{"capability":"Provides utilities for working with Spark DataFrames that have complex nested schemas, including flattening StructType columns, renaming nested fields, and converting JSON schemas to Spark schemas.","skillfed_tags":["spark-dataframe","schema-flattening","abandoned"],"use_cases":["Flatten deeply nested JSON or Parquet data into a single-level schema for simpler SQL queries and easier column access.","Rename nested struct fields programmatically without manually reconstructing the entire schema using struct() or cast().","Convert JSON schema definitions to Spark StructType objects for schema validation or DataFrame creation.","Reduce I/O overhead when reading only specific nested columns from Parquet files by flattening the schema first.","Transform snake_case naming conventions across nested column hierarchies in a single operation."],"what_it_does":"Sparkaid is a utility library for PySpark that simplifies working with DataFrames containing complex nested schemas. It provides functions to flatten StructType columns (removing nesting layers), rename fields within nested structures, and convert JSON schemas to Spark StructType objects. The package addresses common pain points when working with nested data: complex SQL queries, difficulty renaming or casting nested columns, and unnecessary I/O overhead when reading only specific nested columns from Parquet files.\n\nThe library's core feature is its flatten() function, which unpacks nested StructType columns into flat columns with configurable separators (e.g., converting {\"parent\": {\"child\": \"value\"}} into {\"parent_child\": \"value\"}). Version 1.0.0 introduced a breaking change where flatten() now stops at ArrayType columns by default, requiring explicit configuration to unpack arrays. The package depends solely on pyspark and installs as a pure Python wheel with minimal friction.","worth_installing":"No. The package is abandoned (last release August 2022, no commits since September 2022) and carries a copyleft license that may restrict use in proprietary projects. While it addresses a real problem in Spark schema manipulation, modern Spark versions (3.x+) and the Databricks documentation now provide native or better-maintained alternatives. Install only if you are locked into an older Spark version and cannot upgrade."},"id":"sparkaid","links":{"html":"https://skillfed.io/packages/sparkaid","md":"https://skillfed.io/packages/sparkaid.md","pypi":"https://pypi.org/project/sparkaid/"},"maintenance":{"status":"abandoned"},"meta":{"latest_release":"2022-08-29","license_spdx":null,"license_treatment":"copyleft","name":"sparkaid","python_support":"supports_current","summary":"Utils for working with Spark"},"popularity":{"monthly_downloads":191085,"position":9892,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"1.0.0"}