{"categories":[{"label":"Python Modules","url":"https://skillfed.io/packages/category/software-development-libraries-python-modules/15"}],"enrichment":{"capability":"Converts pydantic model definitions to pyarrow schemas, enabling direct schema mapping from validated data models to columnar storage formats.","skillfed_tags":["schema-generation","data-validation","columnar-storage"],"use_cases":["Build a data pipeline that validates incoming records with pydantic, then writes them to parquet using the same schema definition.","Generate pyarrow schemas programmatically from pydantic models to ensure validation rules and storage schemas stay synchronized.","Convert nested pydantic models with complex types (lists, dicts, structs) into equivalent pyarrow struct and map types for analytics.","Exclude certain pydantic fields from the pyarrow schema using pydantic's exclude flag without duplicating schema logic.","Map pydantic Enum and Literal types to pyarrow dictionary encodings for efficient categorical storage."],"what_it_does":"pydantic-to-pyarrow bridges pydantic's data validation layer with pyarrow's columnar storage and analytics capabilities. It reads pydantic model type hints and field constraints to generate equivalent pyarrow schemas, eliminating the need to manually specify schemas separately from validation rules. This is useful in data pipelines where validated data moves into pyarrow tables or parquet files.\n\nThe library handles common Python types (str, int, float, datetime, UUID, Optional, List, Dict, Enum, nested models) and pydantic-specific types (NaiveDatetime, AwareDatetime). It respects pydantic field metadata like exclude flags and serialization aliases. The conversion table documents overflow risks for numeric types and special handling requirements for timezone-aware datetimes and UUIDs.","worth_installing":"Yes, if you use both pydantic for validation and pyarrow for storage or analytics in the same pipeline. The library eliminates schema duplication and keeps validation rules and storage schemas in sync. Install with caution if you depend on PyArrow < 15 with Python 3.9+\u2014manually constrain NumPy 1.x. The aging maintenance status (560 days since last release) is a minor concern but not a blocker; the codebase is stable and the repository remains active."},"id":"pydantic-to-pyarrow","links":{"html":"https://skillfed.io/packages/pydantic-to-pyarrow","md":"https://skillfed.io/packages/pydantic-to-pyarrow.md","pypi":"https://pypi.org/project/pydantic-to-pyarrow/"},"maintenance":{"status":"aging"},"meta":{"latest_release":"2025-01-31","license_spdx":null,"license_treatment":"permissive","name":"pydantic-to-pyarrow","python_support":"supports_current","summary":"Conversion from pydantic models to pyarrow schemas"},"popularity":{"monthly_downloads":235158,"position":9007,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.1.6"}