---
id: s3torchconnectorclient
version: "1.5.0"
license: unclear
license_treatment: permissive
maintenance: active
---

# s3torchconnectorclient — Internal S3 client implementation for s3torchconnector

License: permissive · Maintenance: active · Downloads: 1.4M/mo

## What it is and what it does

s3torchconnectorclient is an internal S3 client implementation that provides optimized access to Amazon S3 for PyTorch training workflows. It supports both map-style datasets for random access and iterable-style datasets for streaming sequential access, handling concurrent S3 requests and bucket listing automatically without requiring custom integration code.

The package includes checkpoint management interfaces for saving and loading model checkpoints directly to S3 without intermediate local storage. For distributed training, it implements PyTorch's StorageWriter and StorageReader interfaces. It supports Python 3.8 through 3.14 and requires PyTorch 2.0 or newer, with PyTorch 2.3 or newer needed for distributed checkpoint features.

Use it for:

- Load training datasets from S3 buckets with automatic optimization for high-throughput data pipelines
- Save model checkpoints directly to S3 during training without writing to local disk first
- Implement distributed checkpointing across multiple training nodes using PyTorch's distributed checkpoint framework
- Stream sequential training data from S3 using iterable-style datasets for memory-efficient training
- Access S3 Express One Zone directory buckets for low-latency training data access

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Internal S3 client implementation providing optimized data loading and checkpoint management for PyTorch training jobs accessing Amazon S3.

Yes, if you are training PyTorch models with data in Amazon S3 and want optimized, managed S3 access without writing custom integration code. Install friction is moderate (platform-specific wheels for Linux and macOS only), but maintenance is active and there are no known vulnerabilities. Verify whether you should install this directly or rely on it as a transitive dependency.

## Install

pip install s3torchconnectorclient
uv add s3torchconnectorclient
poetry add s3torchconnectorclient

## Installing s3torchconnectorclient

Before you install: Medium install friction due to platform-specific wheels for Linux and macOS only; macOS x86_64 support deprecated. Actively maintained with recent releases. No runtime dependencies to manage.

License in practice: Permissive license treatment allows broad use without significant legal constraints.

Quickstart:

pip install s3torchconnectorclient

from s3torchconnectorclient import S3MapDataset

dataset = S3MapDataset.from_prefix("s3://bucket/prefix", region="us-east-1")
item = dataset[0]
print(item.key)

AWS credentials required via EC2 Instance Role, AWS CLI, credential files (~/.aws/credentials), or environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY). PyTorch >= 2.0 required; PyTorch >= 2.3 for distributed checkpoint features. Pre-built wheels available for Linux and macOS only.

Verify before relying:

- Whether this package is intended for direct installation or only as an internal dependency
- Specific performance improvements or throughput gains versus standard S3 access
- Full compatibility matrix with PyTorch versions outside the 2.0+ range

## Package facts

- License: not declared (permissive)
- Python support: supports_current
- Install friction: medium
- Maintenance: active
- Downloads: 1.4M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

s3 pytorch data loading, amazon s3 torch connector, pytorch s3 checkpointing, s3 distributed checkpoints, high throughput s3 training, pytorch s3 datasets, s3 checkpoint management, pytorch-integration, aws-s3, distributed-training

[View on SkillFed](https://skillfed.io/packages/s3torchconnectorclient) · [View on PyPI](https://pypi.org/project/s3torchconnectorclient/)