sparkdantic
A pydantic -> spark schema library
Decision gist · record as of 2026-08-14
Yes, if you work with both Pydantic and PySpark. Low install friction, no security vulnerabilities, and active maintenance make it a safe choice. The optional PySpark dependency is a practical design—you can use JSON schema generation immediately. The main caveat is that license information is unavailable, so verify licensing for your use case before committing to production.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- PySpark 3.3.0 or higher (up to but not including 4.2.0) required for StructType schema generation; JSON schema generation works without PySpark.
- Low install friction with a pure-Python wheel.
- PySpark is optional; JSON schema generation works without it, but StructType conversion requires PySpark 3.3.0 or higher (up to but not including 4.2.0).
License · maintenance · safety
(unclear)
last release 2026-07-13 (32 days)
0 known vulnerabilities (OSV.dev, 2026-08-14) · 1,408,664 downloads/mo, #3,942 on PyPI
Alternatives
Verify before relying
pip install sparkdantic
from sparkdantic import SparkModel
from typing import List
class MyModel(SparkModel):
name: str
age: int
hobbies: List[str]
spark_schema = MyModel.model_spark_schema()- Whether the package is actively maintained beyond the 32-day release window (no repo stars, commit history, or archived status data available).
- Real-world performance and compatibility with edge cases in complex Pydantic type hierarchies.
What it is and what it does
SparkDantic bridges Pydantic and PySpark by letting you define data models once as Pydantic classes and automatically generate PySpark schemas from them. It extends Pydantic's BaseModel with a SparkModel class that adds schema generation methods, and also provides standalone functions to convert existing Pydantic models. The package handles type mapping from Python/Pydantic types to Spark native types, supports Enums (mixed with int or str), and allows field-level type coercion via a SparkField function. PySpark is optional—you can generate JSON-formatted Spark schemas without it, but StructType generation requires PySpark 3.3.0 or higher (up to but not including 4.2.0).
The package depends on jinja2, jmespath, packaging, pydantic, and pyparsing. It's designed for workflows where you need to validate data with Pydantic and then load or process it in Spark, eliminating the need to maintain separate schema definitions. Fields can be excluded from Spark schemas using Pydantic's built-in exclude mechanism, useful for Spark-incompatible types.
Use it for
- Define a Pydantic model for data validation, then generate a PySpark StructType schema for DataFrame operations without manual schema definition.
- Convert existing Pydantic BaseModel classes to Spark schemas using the standalone create_spark_schema function for legacy codebases.
- Generate JSON-formatted Spark schemas from Pydantic models for serialization or schema registry integration without requiring PySpark installed.
- Enforce type safety across validation and Spark processing by coercing Pydantic fields to specific Spark types via SparkField.
- Exclude Spark-incompatible fields (e.g., arbitrary objects) from schema generation while keeping them in the Pydantic model for validation.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you work with both Pydantic and PySpark.
Low install friction, no security vulnerabilities, and active maintenance make it a safe choice. The optional PySpark dependency is a practical design—you can use JSON schema generation immediately. The main caveat is that license information is unavailable, so verify licensing for your use case before committing to production.
Install
sparkdantic on PyPI
Before you install
Low install friction with a pure-Python wheel. PySpark is optional; JSON schema generation works without it, but StructType conversion requires PySpark 3.3.0 or higher (up to but not including 4.2.0). Actively maintained as of 32 days ago.
PySpark 3.3.0 or higher (up to but not including 4.2.0) required for StructType schema generation; JSON schema generation works without PySpark.
Quickstart
pip install sparkdantic
from sparkdantic import SparkModel
from typing import List
class MyModel(SparkModel):
name: str
age: int
hobbies: List[str]
spark_schema = MyModel.model_spark_schema()Verify before relying
- Whether the package is actively maintained beyond the 32-day release window (no repo stars, commit history, or archived status data available).
- Real-world performance and compatibility with edge cases in complex Pydantic type hierarchies.
Package facts
| License | Not declared unclear |
| Python support | Supports the current Python release <4.0.0,>=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 5 packagesjinja2jmespathpackagingpydanticpyparsing |
| Maintenance | Actively maintained 32 days since the last release |
| First released | |
| Downloads | 1,408,664 / month, #3,942 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
Evidence: sparkdantic-2.8.4-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “pydantic sparkdantic”
- sparkdanticConverts Pydantic models to PySpark schemas (StructType or JSON…
- pydantic-extra-typesProvides additional type validators and serializers for Pydantic v2…
- pydantic-compatProvides compatibility mixins and adapters to use either pydantic v1…
Give your agent the search over MCP, or paste the wish link into any chat.
More Application Frameworks packages
FastAPI is a Python web framework for building REST APIs using type hints, with automatic request validation, serialization, and interactive API documentation.
Provides a way to document function parameters, class attributes, return types, and variables inline using Python's `Annotated` type hint syntax instead of traditional docstrings.
Textual is a Python framework for building cross-platform user interfaces that run in the terminal or web browser using a modern, component-based API.
Install it if you're developing CLI tools, dashboards, or interactive terminal applications.
Typer builds command-line applications from Python functions using type hints, automatically generating help text, argument parsing, and shell completion.
Install it if you are building CLIs in Python.
Build and connect to Model Context Protocol servers that expose tools, resources, and prompts to LLM applications over stdio, HTTP, or SSE transports.
Install it if you need to build or connect to servers.
Werkzeug is a WSGI utility library providing request/response objects, URL routing, an interactive debugger, HTTP utilities, and a development server for building web applications.
See also sparkaid · tinsel · pbspark · pydantic-to-typescript · pydantic-spark · graphene-pydantic · pydantic-to-typescript2 · dydantic · pyspark-data-sources · quinn