sparkdantic

What it is and what it does

SparkDantic bridges Pydantic and PySpark by letting you define data models once as Pydantic classes and automatically generate PySpark schemas from them. It extends Pydantic's BaseModel with a SparkModel class that adds schema generation methods, and also provides standalone functions to convert existing Pydantic models. The package handles type mapping from Python/Pydantic types to Spark native types, supports Enums (mixed with int or str), and allows field-level type coercion via a SparkField function. PySpark is optional—you can generate JSON-formatted Spark schemas without it, but StructType generation requires PySpark 3.3.0 or higher (up to but not including 4.2.0).

The package depends on jinja2, jmespath, packaging, pydantic, and pyparsing. It's designed for workflows where you need to validate data with Pydantic and then load or process it in Spark, eliminating the need to maintain separate schema definitions. Fields can be excluded from Spark schemas using Pydantic's built-in exclude mechanism, useful for Spark-incompatible types.

Use it for:

• Define a Pydantic model for data validation, then generate a PySpark StructType schema for DataFrame operations without manual schema definition.
• Convert existing Pydantic BaseModel classes to Spark schemas using the standalone create_spark_schema function for legacy codebases.
• Generate JSON-formatted Spark schemas from Pydantic models for serialization or schema registry integration without requiring PySpark installed.
• Enforce type safety across validation and Spark processing by coercing Pydantic fields to specific Spark types via SparkField.
• Exclude Spark-incompatible fields (e.g., arbitrary objects) from schema generation while keeping them in the Pydantic model for validation.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Converts Pydantic models to PySpark schemas (StructType or JSON format), allowing you to define data structures once and generate compatible Spark schemas automatically.

Yes, if you work with both Pydantic and PySpark. Low install friction, no security vulnerabilities, and active maintenance make it a safe choice. The optional PySpark dependency is a practical design—you can use JSON schema generation immediately. The main caveat is that license information is unavailable, so verify licensing for your use case before committing to production.

Install

sparkdantic on PyPI

pip install sparkdantic

uv add sparkdantic

poetry add sparkdantic

Installing sparkdantic

Before you install

Low install friction with a pure-Python wheel. PySpark is optional; JSON schema generation works without it, but StructType conversion requires PySpark 3.3.0 or higher (up to but not including 4.2.0). Actively maintained as of 32 days ago.

Quickstart

pip install sparkdantic

from sparkdantic import SparkModel
from typing import List

class MyModel(SparkModel):
    name: str
    age: int
    hobbies: List[str]

spark_schema = MyModel.model_spark_schema()

PySpark 3.3.0 or higher (up to but not including 4.2.0) required for StructType schema generation; JSON schema generation works without PySpark.

Verify before relying

• Whether the package is actively maintained beyond the 32-day release window (no repo stars, commit history, or archived status data available).
• Real-world performance and compatibility with edge cases in complex Pydantic type hierarchies.

Package facts

Evidence: sparkdantic-2.8.4-py3-none-any.whl

Tags

More Application Frameworks packages