litdata

What it is and what it does

LitData is a data pipeline library for PyTorch that addresses the bottleneck of slow data loading during model training. It provides two main workflows: StreamingRawDataset for streaming raw files directly from cloud or local storage with no preprocessing, and an optimize-then-stream pattern that chunks datasets into efficient binary formats for maximum throughput. The library handles asynchronous batched downloads with built-in retries, supports S3, GCS, Azure, and local paths through a unified interface, and integrates with PyTorch's DataLoader.

The package is built on torch, torchvision, and lightning-utilities, with additional cloud and file I/O support via boto3, requests, obstore, and tifffile. It targets developers working with large datasets that either don't fit locally, consist of millions of small files, or stream slowly from the cloud. The optimize step is a one-time preprocessing task that transforms raw data into chunked binary format; the streaming step then loads that data on-demand during training without requiring full local copies.

Use it for:

• Stream image datasets from S3 during training without downloading the entire dataset to local disk.
• Preprocess and optimize a large dataset once, then train multiple models against the optimized version from cloud storage.
• Load datasets consisting of millions of small files (e.g., individual images or audio clips) efficiently via batched async downloads.
• Resume training from a checkpoint while continuing to stream data, with the index cached locally for fast restarts.
• Distribute data processing across workers before optimization to create embeddings or apply transformations at scale.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

LitData optimizes and streams large datasets for PyTorch training by chunking data into efficient formats and loading directly from cloud storage or local paths without full local downloads.

Yes, if you train PyTorch models on datasets larger than local storage or consisting of many small cloud files. The library is actively maintained, has no known vulnerabilities, and low install friction. The permissive Apache-2.0 license poses no restrictions. Start with StreamingRawDataset for immediate benefit; use the optimize workflow when I/O becomes the bottleneck. Not necessary for small local datasets that fit in memory or on disk.

Install

litdata on PyPI

pip install litdata

uv add litdata

poetry add litdata

Installing litdata

Before you install

Low friction; pure Python wheel with no compiled dependencies. Active maintenance (last commit 2026-08-14) and recent release cycle. Requires torch and torchvision as runtime dependencies, which are typically already present in PyTorch projects.

License in practice

Apache-2.0 permissive license allows commercial and private use with minimal restrictions; suitable for most production and research contexts.

Quickstart

pip install litdata

from litdata import StreamingRawDataset
from torch.utils.data import DataLoader

dataset = StreamingRawDataset("s3://my-bucket/images/")
loader = DataLoader(dataset, batch_size=32)
for batch in loader:
    pass  # train_step(batch)

Requires Python >=3.10 and torch installed. For cloud paths (S3, GCS, Azure), appropriate cloud credentials and boto3 must be configured.

Verify before relying

• Whether the claimed 20x training speedup is measured under specific dataset/hardware conditions and how generalizable that figure is.
• Whether uvloop (optional on Linux/macOS) provides meaningful performance gains for typical workloads.
• Whether polars is required for keyed lookup and in-place patches (description cuts off mid-sentence).

Package facts

Evidence: litdata-0.2.68-py3-none-any.whl

Keywords: deep learning, pytorch, AI, streaming, cloud, data processing

Tags

More Artificial Intelligence packages