{"categories":[{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/7"}],"enrichment":{"capability":"Petastorm enables direct training of deep learning models from Apache Parquet datasets using TensorFlow, PyTorch, PySpark, or pure Python, with support for selective column access, shuffling, filtering, and distributed parallelism.","skillfed_tags":["data-loading","parquet","distributed-training"],"use_cases":["Train TensorFlow or PyTorch models on large Parquet datasets stored on HDFS or local filesystems without loading entire datasets into memory.","Generate Parquet datasets from raw data using PySpark, then iterate over them in Python ML training loops with automatic schema validation.","Distribute training across multiple GPUs by partitioning Petastorm datasets and reading different partitions on different workers.","Apply row-level filtering and selective column reads to reduce I/O when training on subsets of large datasets.","Compress image and array data using standard codecs (JPEG, PNG) or custom codecs within a single Parquet-backed dataset."],"what_it_does":"Petastorm is a data access library that bridges Apache Parquet storage and popular Python machine learning frameworks. It was developed at Uber ATG to enable efficient, distributed training of deep learning models directly from Parquet datasets without requiring intermediate format conversions. The library handles the schema mapping between Parquet and framework-specific types (TensorFlow, PyTorch, PySpark), and provides a unified reader interface that supports selective column access, row filtering, shuffling, and partitioning for multi-GPU training.\n\nYou use Petastorm in two phases: first, generate a Parquet dataset using PySpark with the Unischema API to define field types, shapes, and compression codecs; then, read from that dataset using the Reader class or framework-specific adapters (tf_tensors for TensorFlow, DataLoader for PyTorch). The library handles parallelism internally via threads, processes, or single-threaded modes, and supports local caching to reduce repeated I/O.","worth_installing":"Yes, if you are already using Parquet for data storage and training with TensorFlow, PyTorch, or PySpark. The low install friction and permissive license make it a straightforward addition. However, note the aging maintenance status (224 days since last release)\u2014verify compatibility with your specific framework versions before committing to production use. No known security vulnerabilities."},"id":"petastorm","links":{"html":"https://skillfed.io/packages/petastorm","md":"https://skillfed.io/packages/petastorm.md","pypi":"https://pypi.org/project/petastorm/"},"maintenance":{"status":"aging"},"meta":{"latest_release":"2026-01-02","license_spdx":null,"license_treatment":"permissive","name":"petastorm","python_support":"supports_current","summary":"Petastorm is a library enabling the use of Parquet storage from Tensorflow, Pytorch, and other Python-based ML training frameworks."},"popularity":{"monthly_downloads":199587,"position":9705,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.13.1"}