{"categories":[{"label":"Application Frameworks","url":"https://skillfed.io/packages/category/software-development-libraries-application-frameworks/2"}],"enrichment":{"capability":"Converts Pydantic models to PySpark schemas (StructType or JSON format), allowing you to define data structures once and generate compatible Spark schemas automatically.","skillfed_tags":["pydantic-integration","spark-schema-generation","data-validation"],"use_cases":["Define a Pydantic model for data validation, then generate a PySpark StructType schema for DataFrame operations without manual schema definition.","Convert existing Pydantic BaseModel classes to Spark schemas using the standalone create_spark_schema function for legacy codebases.","Generate JSON-formatted Spark schemas from Pydantic models for serialization or schema registry integration without requiring PySpark installed.","Enforce type safety across validation and Spark processing by coercing Pydantic fields to specific Spark types via SparkField.","Exclude Spark-incompatible fields (e.g., arbitrary objects) from schema generation while keeping them in the Pydantic model for validation."],"what_it_does":"SparkDantic bridges Pydantic and PySpark by letting you define data models once as Pydantic classes and automatically generate PySpark schemas from them. It extends Pydantic's BaseModel with a SparkModel class that adds schema generation methods, and also provides standalone functions to convert existing Pydantic models. The package handles type mapping from Python/Pydantic types to Spark native types, supports Enums (mixed with int or str), and allows field-level type coercion via a SparkField function. PySpark is optional\u2014you can generate JSON-formatted Spark schemas without it, but StructType generation requires PySpark 3.3.0 or higher (up to but not including 4.2.0).\n\nThe package depends on jinja2, jmespath, packaging, pydantic, and pyparsing. It's designed for workflows where you need to validate data with Pydantic and then load or process it in Spark, eliminating the need to maintain separate schema definitions. Fields can be excluded from Spark schemas using Pydantic's built-in exclude mechanism, useful for Spark-incompatible types.","worth_installing":"Yes, if you work with both Pydantic and PySpark. Low install friction, no security vulnerabilities, and active maintenance make it a safe choice. The optional PySpark dependency is a practical design\u2014you can use JSON schema generation immediately. The main caveat is that license information is unavailable, so verify licensing for your use case before committing to production."},"id":"sparkdantic","links":{"html":"https://skillfed.io/packages/sparkdantic","md":"https://skillfed.io/packages/sparkdantic.md","pypi":"https://pypi.org/project/sparkdantic/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-07-13","license_spdx":null,"license_treatment":"unclear","name":"sparkdantic","python_support":"supports_current","summary":"A pydantic -> spark schema library"},"popularity":{"monthly_downloads":1408664,"position":3942,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"2.8.4"}