s3torchconnector

What it is and what it does

s3torchconnector bridges PyTorch's data loading pipeline with Amazon S3, eliminating the need to write custom code for bucket listing and concurrent request handling. It implements PyTorch's dataset primitives—both map-style for random access and iterable-style for streaming—so training jobs can fetch data directly from S3 with automatic performance optimization. It also provides checkpoint interfaces to save and load model state directly to S3 without intermediate local storage, and includes distributed checkpoint support via StorageWriter and StorageReader implementations compatible with PyTorch's distributed checkpoint framework.

The package depends on torch and s3torchconnectorclient (a compiled native component), which creates platform-specific installation requirements. It supports Python 3.8 through 3.14 and requires PyTorch 2.0 or newer (PyTorch 2.3+ for distributed checkpoint features). AWS credentials must be configured via EC2 instance role, AWS CLI, credential files, or environment variables before use.

Use it for:

• Stream training data from S3 buckets into PyTorch DataLoaders without downloading to local disk first
• Save model checkpoints directly to S3 during distributed training without staging through local storage
• Load distributed checkpoints from S3 using optimized readers for faster recovery in multi-node training
• Access S3 Express One Zone directory buckets for high-performance data access in time-sensitive training jobs
• Implement random-access training datasets backed by S3 for flexible sampling patterns

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides PyTorch dataset primitives and checkpoint interfaces for efficient data loading and model checkpointing directly from Amazon S3, supporting both map-style and iterable-style datasets.

Yes, if you train PyTorch models on AWS and need to load data from or checkpoint to S3. High install friction (compiled deps, platform-specific wheels) and AWS credential configuration requirements are real costs, but the package is actively maintained, permissively licensed, and eliminates custom S3 integration code. Not worth installing if you're not on Linux/macOS or don't use S3 for training data.

Install

s3torchconnector on PyPI

pip install s3torchconnector

uv add s3torchconnector

poetry add s3torchconnector

Installing s3torchconnector

Before you install

High install friction due to compiled dependencies (s3torchconnectorclient) and platform-specific wheels; pre-built wheels available only for Linux and macOS. Maintenance status is active with a recent release.

License in practice

Permissive license treatment allows commercial and private use without restriction.

Quickstart

pip install s3torchconnector

from s3torchconnector import S3IterableDataset

dataset = S3IterableDataset.from_prefix(
    "s3://my-bucket/data",
    region="us-east-1"
)
for item in dataset:
    print(item.key, len(item.read()))

AWS credentials must be configured (EC2 role, AWS CLI, ~/.aws/credentials, or AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY environment variables). Pre-built wheels available only for Linux and macOS; other platforms require building from source.

Verify before relying

• Whether macOS x86_64 wheel support deprecation affects your deployment target
• Python 3.8 support timeline and deprecation schedule for your long-term maintenance
• Performance gains over manual S3 listing and concurrent request management in your workload

Package facts

Evidence: s3torchconnector-1.5.0.tar.gz

Tags

More Utilities packages