s3torchconnectorclient

What it is and what it does

s3torchconnectorclient is an internal S3 client implementation that provides optimized access to Amazon S3 for PyTorch training workflows. It supports both map-style datasets for random access and iterable-style datasets for streaming sequential access, handling concurrent S3 requests and bucket listing automatically without requiring custom integration code.

The package includes checkpoint management interfaces for saving and loading model checkpoints directly to S3 without intermediate local storage. For distributed training, it implements PyTorch's StorageWriter and StorageReader interfaces. It supports Python 3.8 through 3.14 and requires PyTorch 2.0 or newer, with PyTorch 2.3 or newer needed for distributed checkpoint features.

Use it for:

• Load training datasets from S3 buckets with automatic optimization for high-throughput data pipelines
• Save model checkpoints directly to S3 during training without writing to local disk first
• Implement distributed checkpointing across multiple training nodes using PyTorch's distributed checkpoint framework
• Stream sequential training data from S3 using iterable-style datasets for memory-efficient training
• Access S3 Express One Zone directory buckets for low-latency training data access

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Internal S3 client implementation providing optimized data loading and checkpoint management for PyTorch training jobs accessing Amazon S3.

Yes, if you are training PyTorch models with data in Amazon S3 and want optimized, managed S3 access without writing custom integration code. Install friction is moderate (platform-specific wheels for Linux and macOS only), but maintenance is active and there are no known vulnerabilities. Verify whether you should install this directly or rely on it as a transitive dependency.

Install

s3torchconnectorclient on PyPI

pip install s3torchconnectorclient

uv add s3torchconnectorclient

poetry add s3torchconnectorclient

Installing s3torchconnectorclient

Before you install

Medium install friction due to platform-specific wheels for Linux and macOS only; macOS x86_64 support deprecated. Actively maintained with recent releases. No runtime dependencies to manage.

License in practice

Permissive license treatment allows broad use without significant legal constraints.

Quickstart

pip install s3torchconnectorclient

from s3torchconnectorclient import S3MapDataset

dataset = S3MapDataset.from_prefix("s3://bucket/prefix", region="us-east-1")
item = dataset[0]
print(item.key)

AWS credentials required via EC2 Instance Role, AWS CLI, credential files (~/.aws/credentials), or environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY). PyTorch >= 2.0 required; PyTorch >= 2.3 for distributed checkpoint features. Pre-built wheels available for Linux and macOS only.

Verify before relying

• Whether this package is intended for direct installation or only as an internal dependency
• Specific performance improvements or throughput gains versus standard S3 access
• Full compatibility matrix with PyTorch versions outside the 2.0+ range

Package facts

Evidence: s3torchconnectorclient-1.5.0-cp310-cp310-macosx_10_12_x86_64.whl; s3torchconnectorclient-1.5.0-cp310-cp310-macosx_11_0_arm64.whl; s3torchconnectorclient-1.5.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; s3torchconnectorclient-1.5.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; s3torchconnectorclient-1.5.0-cp311-cp311-macosx_10_12_x86_64.whl; s3torchconnectorclient-1.5.0-cp311-cp311-macosx_11_0_arm64.whl; s3torchconnectorclient-1.5.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; s3torchconnectorclient-1.5.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; s3torchconnectorclient-1.5.0-cp312-cp312-macosx_10_13_x86_64.whl; s3torchconnectorclient-1.5.0-cp312-cp312-macosx_11_0_arm64.whl; s3torchconnectorclient-1.5.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; s3torchconnectorclient-1.5.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; s3torchconnectorclient-1.5.0-cp313-cp313-macosx_11_0_arm64.whl; s3torchconnectorclient-1.5.0-cp313-cp313-manylinux_2_28_aarch64.whl; s3torchconnectorclient-1.5.0-cp313-cp313-manylinux_2_28_x86_64.whl; s3torchconnectorclient-1.5.0-cp314-cp314-macosx_11_0_arm64.whl; s3torchconnectorclient-1.5.0-cp314-cp314-manylinux_2_28_aarch64.whl; s3torchconnectorclient-1.5.0-cp314-cp314-manylinux_2_28_x86_64.whl; s3torchconnectorclient-1.5.0-cp38-cp38-macosx_10_12_x86_64.whl; s3torchconnectorclient-1.5.0-cp38-cp38-macosx_11_0_arm64.whl

Tags

More Utilities packages