pydantic-spark

What it is and what it does

pydantic-spark is a lightweight bridge between Pydantic's Python type system and Apache Spark's schema representation. It lets you define data models as Pydantic classes and automatically generate the corresponding Spark schema dictionaries, or reverse the process by generating Python code from an existing Spark schema. The library extends Pydantic's BaseModel through a SparkBase class and adds a spark_schema() method that outputs a schema-compatible dictionary.

The package is designed for data engineers and Python developers working with Spark who want to avoid manually writing schema definitions or keeping Pydantic models and Spark schemas in sync. It includes a coerce_type option for field-level type conversion during schema generation. With only pydantic as a runtime dependency, installation is straightforward, though the project is currently dormant—last updated in late 2023—so new features or maintenance are unlikely.

Use it for:

• Define Spark DataFrame schemas using Pydantic classes, ensuring type safety and validation in Python before writing to Spark.
• Generate boilerplate Spark schema code from existing Pydantic models to reduce manual schema definition work.
• Reverse-engineer Python Pydantic classes from Spark schemas to keep data contracts synchronized across systems.
• Apply field-level type coercion rules during schema generation when Pydantic types need to map to different Spark types.
• Validate and document data pipelines by using Pydantic's validation alongside Spark's schema enforcement.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Converts Pydantic class definitions to Apache Spark schemas and generates Python code from Spark schemas, bridging type validation and distributed data processing.

Yes, if you actively use both Pydantic and Spark and want to avoid duplicating schema definitions. The low install friction and permissive license make it a low-risk addition. However, be aware that the project is dormant—no updates since late 2023—so you should verify compatibility with your current Pydantic and Spark versions before relying on it in production, and plan to maintain a fork if critical bugs emerge.

Install

pydantic-spark on PyPI

pip install pydantic-spark

uv add pydantic-spark

poetry add pydantic-spark

Installing pydantic-spark

Before you install

Low install friction with a single runtime dependency on pydantic. Maintenance is dormant—last commit was 2024-03-04 and no releases since 2023-11-24—so expect no active bug fixes or feature development, though the codebase remains archived and available.

License in practice

MIT license is permissive, allowing commercial and private use with minimal restrictions; you may use this package freely in most contexts.

Quickstart

from pydantic_spark.base import SparkBase

class TestModel(SparkBase):
    key1: str
    key2: int

schema_dict = TestModel.spark_schema()

Requires pydantic as a runtime dependency; Spark itself is not listed as a dependency, so you must have it available in your environment separately.

Verify before relying

• Whether Spark is an implicit peer dependency or truly optional at runtime.
• Compatibility with recent Pydantic v2 major version changes and their breaking API shifts.
• Whether the coerce_type feature and other advanced options are production-ready or experimental.

Package facts

Evidence: pydantic_spark-1.0.1-py3-none-any.whl

Keywords: pydantic, spark

Tags

More Database packages