---
id: mosaicml-streaming
version: "0.13.0"
license: unclear
license_treatment: unclear
maintenance: active
---

# mosaicml-streaming — Streaming lets users create PyTorch compatible datasets that can be streamed from cloud-based object stores

License: unclear · Maintenance: active · Downloads: 928.7K/mo

## What it is and what it does

StreamingDataset is a PyTorch data loader designed to efficiently fetch training data from cloud object storage on-demand, rather than downloading entire datasets upfront. It acts as a drop-in replacement for PyTorch's IterableDataset, integrating with AWS S3, Google Cloud Storage, Azure Blob Storage, OCI Object Storage, and S3-compatible services. The package handles data serialization in MDS (Mosaic Data Shard) format, CSV, TSV, or JSONL, with optional compression via zstd, Brotli, or snappy.

The core value proposition is enabling distributed multi-node training without requiring all data to be co-located with compute. It guarantees deterministic sample ordering across any number of GPUs, nodes, or workers—a feature that aids reproducibility and debugging. The package supports seamless mixing of multiple datasets via the Stream abstraction, allowing proportional or absolute sampling strategies. It is built for large-scale model training workflows where data volume and cloud infrastructure costs are primary concerns.

Use it for:

- Train vision models on large image datasets stored in S3 or GCS without local disk staging.
- Stream multimodal data (text + images, video + captions) from cloud storage for foundation model pretraining.
- Mix multiple datasets on-the-fly during training with deterministic reproducibility across training runs.
- Distribute training across multiple nodes where each worker streams only the samples it needs, reducing network and storage bottlenecks.
- Convert and persist raw datasets into sharded MDS format for efficient repeated training runs.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Streams large training datasets from cloud storage (AWS S3, GCS, Azure, OCI) as a PyTorch-compatible IterableDataset, handling images, text, video, and multimodal data with built-in compression and deterministic shuffling.

Yes, if you train on large datasets in cloud storage and need deterministic, distributed data loading. The active maintenance, low install friction, and PyTorch integration make it a solid choice. However, verify the license terms first (currently marked unclear in metadata), and confirm that the 18 runtime dependencies align with your environment. Not necessary if your data fits locally or if you use a different data pipeline framework.

## Install

pip install mosaicml-streaming
uv add mosaicml-streaming
poetry add mosaicml-streaming

## Installing mosaicml-streaming

Before you install: Low friction: pure Python wheel with no compiled dependencies. Active maintenance (last commit 2026-06-25, 1545 stars). However, 18 runtime dependencies including torch, torchvision, and transformers mean a substantial environment footprint if not already present.

License in practice: License treatment is unclear—no SPDX identifier or raw license text in metadata. Verify the actual license terms in the repository before adopting in proprietary or restricted-license contexts.

Quickstart:

pip install mosaicml-streaming

from streaming import StreamingDataset
from torch.utils.data import DataLoader

dataset = StreamingDataset(local='/tmp/cache', remote='s3://bucket/data', shuffle=True)
loader = DataLoader(dataset, batch_size=32)
for batch in loader:
    pass  # train

Requires Python >=3.10 and torch installed. Cloud credentials (AWS, GCS, Azure, OCI) must be configured in your environment for remote access.

Verify before relying:

- Actual license terms and restrictions (metadata shows 'unclear' treatment)
- Whether all 18 runtime dependencies are truly required or optional/conditional
- Performance characteristics and throughput guarantees for different cloud providers
- Specific dataset mixing proportions and sampling behavior in production scenarios

## Package facts

- License: not declared (unclear)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 928.7K/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

pytorch streaming dataset cloud storage, distributed training data loading, cloud object store dataset, large scale ml data pipeline, s3 gcs azure dataset loader, multimodal data streaming, deterministic distributed sampling, distributed-training, cloud-storage, pytorch-data

[View on SkillFed](https://skillfed.io/packages/mosaicml-streaming) · [View on PyPI](https://pypi.org/project/mosaicml-streaming/)