{"categories":[{"label":"Database","url":"https://skillfed.io/packages/category/database/4"}],"enrichment":{"capability":"Converts Pydantic class definitions to Apache Spark schemas and generates Python code from Spark schemas, bridging type validation and distributed data processing.","skillfed_tags":["schema-generation","spark-integration","data-validation"],"use_cases":["Define Spark DataFrame schemas using Pydantic classes, ensuring type safety and validation in Python before writing to Spark.","Generate boilerplate Spark schema code from existing Pydantic models to reduce manual schema definition work.","Reverse-engineer Python Pydantic classes from Spark schemas to keep data contracts synchronized across systems.","Apply field-level type coercion rules during schema generation when Pydantic types need to map to different Spark types.","Validate and document data pipelines by using Pydantic's validation alongside Spark's schema enforcement."],"what_it_does":"pydantic-spark is a lightweight bridge between Pydantic's Python type system and Apache Spark's schema representation. It lets you define data models as Pydantic classes and automatically generate the corresponding Spark schema dictionaries, or reverse the process by generating Python code from an existing Spark schema. The library extends Pydantic's BaseModel through a SparkBase class and adds a spark_schema() method that outputs a schema-compatible dictionary.\n\nThe package is designed for data engineers and Python developers working with Spark who want to avoid manually writing schema definitions or keeping Pydantic models and Spark schemas in sync. It includes a coerce_type option for field-level type conversion during schema generation. With only pydantic as a runtime dependency, installation is straightforward, though the project is currently dormant\u2014last updated in late 2023\u2014so new features or maintenance are unlikely.","worth_installing":"Yes, if you actively use both Pydantic and Spark and want to avoid duplicating schema definitions. The low install friction and permissive license make it a low-risk addition. However, be aware that the project is dormant\u2014no updates since late 2023\u2014so you should verify compatibility with your current Pydantic and Spark versions before relying on it in production, and plan to maintain a fork if critical bugs emerge."},"id":"pydantic-spark","links":{"html":"https://skillfed.io/packages/pydantic-spark","md":"https://skillfed.io/packages/pydantic-spark.md","pypi":"https://pypi.org/project/pydantic-spark/"},"maintenance":{"status":"dormant"},"meta":{"latest_release":"2023-11-24","license_spdx":null,"license_treatment":"permissive","name":"pydantic-spark","python_support":"supports_current","summary":"Converting pydantic classes to spark schemas"},"popularity":{"monthly_downloads":199617,"position":9702,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"1.0.1"}