mosaicml-streaming

What it is and what it does

StreamingDataset is a PyTorch data loader designed to efficiently fetch training data from cloud object storage on-demand, rather than downloading entire datasets upfront. It acts as a drop-in replacement for PyTorch's IterableDataset, integrating with AWS S3, Google Cloud Storage, Azure Blob Storage, OCI Object Storage, and S3-compatible services. The package handles data serialization in MDS (Mosaic Data Shard) format, CSV, TSV, or JSONL, with optional compression via zstd, Brotli, or snappy.

The core value proposition is enabling distributed multi-node training without requiring all data to be co-located with compute. It guarantees deterministic sample ordering across any number of GPUs, nodes, or workers—a feature that aids reproducibility and debugging. The package supports seamless mixing of multiple datasets via the Stream abstraction, allowing proportional or absolute sampling strategies. It is built for large-scale model training workflows where data volume and cloud infrastructure costs are primary concerns.

Use it for:

• Train vision models on large image datasets stored in S3 or GCS without local disk staging.
• Stream multimodal data (text + images, video + captions) from cloud storage for foundation model pretraining.
• Mix multiple datasets on-the-fly during training with deterministic reproducibility across training runs.
• Distribute training across multiple nodes where each worker streams only the samples it needs, reducing network and storage bottlenecks.
• Convert and persist raw datasets into sharded MDS format for efficient repeated training runs.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Streams large training datasets from cloud storage (AWS S3, GCS, Azure, OCI) as a PyTorch-compatible IterableDataset, handling images, text, video, and multimodal data with built-in compression and deterministic shuffling.

Yes, if you train on large datasets in cloud storage and need deterministic, distributed data loading. The active maintenance, low install friction, and PyTorch integration make it a solid choice. However, verify the license terms first (currently marked unclear in metadata), and confirm that the 18 runtime dependencies align with your environment. Not necessary if your data fits locally or if you use a different data pipeline framework.

Install

mosaicml-streaming on PyPI

pip install mosaicml-streaming

uv add mosaicml-streaming

poetry add mosaicml-streaming

Installing mosaicml-streaming

Before you install

Low friction: pure Python wheel with no compiled dependencies. Active maintenance (last commit 2026-06-25, 1545 stars). However, 18 runtime dependencies including torch, torchvision, and transformers mean a substantial environment footprint if not already present.

License in practice

License treatment is unclear—no SPDX identifier or raw license text in metadata. Verify the actual license terms in the repository before adopting in proprietary or restricted-license contexts.

Quickstart

pip install mosaicml-streaming

from streaming import StreamingDataset
from torch.utils.data import DataLoader

dataset = StreamingDataset(local='/tmp/cache', remote='s3://bucket/data', shuffle=True)
loader = DataLoader(dataset, batch_size=32)
for batch in loader:
    pass  # train

Requires Python >=3.10 and torch installed. Cloud credentials (AWS, GCS, Azure, OCI) must be configured in your environment for remote access.

Verify before relying

• Actual license terms and restrictions (metadata shows 'unclear' treatment)
• Whether all 18 runtime dependencies are truly required or optional/conditional
• Performance characteristics and throughput guarantees for different cloud providers
• Specific dataset mixing proportions and sampling behavior in production scenarios

Package facts

Evidence: mosaicml_streaming-0.13.0-py3-none-any.whl

Tags

More Artificial Intelligence packages