{"categories":[{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/7"},{"label":"Information Analysis","url":"https://skillfed.io/packages/category/scientific-engineering-information-analysis/2"}],"enrichment":{"capability":"LitData optimizes and streams large datasets for PyTorch training by chunking data into efficient formats and loading directly from cloud storage or local paths without full local downloads.","skillfed_tags":["pytorch-data-loading","cloud-storage-streaming","distributed-training"],"use_cases":["Stream image datasets from S3 during training without downloading the entire dataset to local disk.","Preprocess and optimize a large dataset once, then train multiple models against the optimized version from cloud storage.","Load datasets consisting of millions of small files (e.g., individual images or audio clips) efficiently via batched async downloads.","Resume training from a checkpoint while continuing to stream data, with the index cached locally for fast restarts.","Distribute data processing across workers before optimization to create embeddings or apply transformations at scale."],"what_it_does":"LitData is a data pipeline library for PyTorch that addresses the bottleneck of slow data loading during model training. It provides two main workflows: StreamingRawDataset for streaming raw files directly from cloud or local storage with no preprocessing, and an optimize-then-stream pattern that chunks datasets into efficient binary formats for maximum throughput. The library handles asynchronous batched downloads with built-in retries, supports S3, GCS, Azure, and local paths through a unified interface, and integrates with PyTorch's DataLoader.\n\nThe package is built on torch, torchvision, and lightning-utilities, with additional cloud and file I/O support via boto3, requests, obstore, and tifffile. It targets developers working with large datasets that either don't fit locally, consist of millions of small files, or stream slowly from the cloud. The optimize step is a one-time preprocessing task that transforms raw data into chunked binary format; the streaming step then loads that data on-demand during training without requiring full local copies.","worth_installing":"Yes, if you train PyTorch models on datasets larger than local storage or consisting of many small cloud files. The library is actively maintained, has no known vulnerabilities, and low install friction. The permissive Apache-2.0 license poses no restrictions. Start with StreamingRawDataset for immediate benefit; use the optimize workflow when I/O becomes the bottleneck. Not necessary for small local datasets that fit in memory or on disk."},"id":"litdata","links":{"html":"https://skillfed.io/packages/litdata","md":"https://skillfed.io/packages/litdata.md","pypi":"https://pypi.org/project/litdata/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-14","license_spdx":null,"license_treatment":"permissive","name":"litdata","python_support":"supports_current","summary":"The Deep Learning framework to train, deploy, and ship AI products Lightning fast."},"popularity":{"monthly_downloads":233560,"position":9037,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.2.68"}