{"categories":[{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/4"}],"enrichment":{"capability":"Streams large training datasets from cloud storage (AWS S3, GCS, Azure, OCI) as a PyTorch-compatible IterableDataset, handling images, text, video, and multimodal data with built-in compression and deterministic shuffling.","skillfed_tags":["distributed-training","cloud-storage","pytorch-data"],"use_cases":["Train vision models on large image datasets stored in S3 or GCS without local disk staging.","Stream multimodal data (text + images, video + captions) from cloud storage for foundation model pretraining.","Mix multiple datasets on-the-fly during training with deterministic reproducibility across training runs.","Distribute training across multiple nodes where each worker streams only the samples it needs, reducing network and storage bottlenecks.","Convert and persist raw datasets into sharded MDS format for efficient repeated training runs."],"what_it_does":"StreamingDataset is a PyTorch data loader designed to efficiently fetch training data from cloud object storage on-demand, rather than downloading entire datasets upfront. It acts as a drop-in replacement for PyTorch's IterableDataset, integrating with AWS S3, Google Cloud Storage, Azure Blob Storage, OCI Object Storage, and S3-compatible services. The package handles data serialization in MDS (Mosaic Data Shard) format, CSV, TSV, or JSONL, with optional compression via zstd, Brotli, or snappy.\n\nThe core value proposition is enabling distributed multi-node training without requiring all data to be co-located with compute. It guarantees deterministic sample ordering across any number of GPUs, nodes, or workers\u2014a feature that aids reproducibility and debugging. The package supports seamless mixing of multiple datasets via the Stream abstraction, allowing proportional or absolute sampling strategies. It is built for large-scale model training workflows where data volume and cloud infrastructure costs are primary concerns.","worth_installing":"Yes, if you train on large datasets in cloud storage and need deterministic, distributed data loading. The active maintenance, low install friction, and PyTorch integration make it a solid choice. However, verify the license terms first (currently marked unclear in metadata), and confirm that the 18 runtime dependencies align with your environment. Not necessary if your data fits locally or if you use a different data pipeline framework."},"id":"mosaicml-streaming","links":{"html":"https://skillfed.io/packages/mosaicml-streaming","md":"https://skillfed.io/packages/mosaicml-streaming.md","pypi":"https://pypi.org/project/mosaicml-streaming/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2025-07-15","license_spdx":null,"license_treatment":"unclear","name":"mosaicml-streaming","python_support":"supports_current","summary":"Streaming lets users create PyTorch compatible datasets that can be streamed from cloud-based object stores"},"popularity":{"monthly_downloads":928673,"position":4707,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"0.13.0"}