pydantic-to-pyarrow

What it is and what it does

pydantic-to-pyarrow bridges pydantic's data validation layer with pyarrow's columnar storage and analytics capabilities. It reads pydantic model type hints and field constraints to generate equivalent pyarrow schemas, eliminating the need to manually specify schemas separately from validation rules. This is useful in data pipelines where validated data moves into pyarrow tables or parquet files.

The library handles common Python types (str, int, float, datetime, UUID, Optional, List, Dict, Enum, nested models) and pydantic-specific types (NaiveDatetime, AwareDatetime). It respects pydantic field metadata like exclude flags and serialization aliases. The conversion table documents overflow risks for numeric types and special handling requirements for timezone-aware datetimes and UUIDs.

Use it for:

• Build a data pipeline that validates incoming records with pydantic, then writes them to parquet using the same schema definition.
• Generate pyarrow schemas programmatically from pydantic models to ensure validation rules and storage schemas stay synchronized.
• Convert nested pydantic models with complex types (lists, dicts, structs) into equivalent pyarrow struct and map types for analytics.
• Exclude certain pydantic fields from the pyarrow schema using pydantic's exclude flag without duplicating schema logic.
• Map pydantic Enum and Literal types to pyarrow dictionary encodings for efficient categorical storage.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Converts pydantic model definitions to pyarrow schemas, enabling direct schema mapping from validated data models to columnar storage formats.

Yes, if you use both pydantic for validation and pyarrow for storage or analytics in the same pipeline. The library eliminates schema duplication and keeps validation rules and storage schemas in sync. Install with caution if you depend on PyArrow < 15 with Python 3.9+—manually constrain NumPy 1.x. The aging maintenance status (560 days since last release) is a minor concern but not a blocker; the codebase is stable and the repository remains active.

Install

pydantic-to-pyarrow on PyPI

pip install pydantic-to-pyarrow

uv add pydantic-to-pyarrow

poetry add pydantic-to-pyarrow

Installing pydantic-to-pyarrow

Before you install

Low friction install with four stable runtime dependencies. Maintenance status is aging—last release was 560 days ago—but the repository remains active with a recent commit on 2025-05-10 and no archived status.

License in practice

MIT license (permissive) places no restrictions on use, modification, or distribution in proprietary or open-source projects.

Quickstart

pip install pydantic-to-pyarrow

from pydantic import BaseModel
from pydantic_to_pyarrow import get_pyarrow_schema

class MyModel(BaseModel):
    name: str
    age: int

schema = get_pyarrow_schema(MyModel)

PyArrow versions < 15 require NumPy 1.x; if forced to use PyArrow < 15 on Python 3.9+, manually constrain NumPy to 1.x to avoid module compatibility errors.

Verify before relying

• Whether overflow handling for int64, uint64, and float64 types is acceptable for your data ranges without manual schema specification.
• Support status and timeline for UUID handling with pyarrow >= 18.0, given the workaround requirement for model_dump() serialization.

Package facts

Evidence: pydantic_to_pyarrow-0.1.6-py3-none-any.whl

Tags

More Python Modules packages