s3torchconnectorclient
Internal S3 client implementation for s3torchconnector
Decision gist · record as of 2026-08-14
Yes, if you are training PyTorch models with data in Amazon S3 and want optimized, managed S3 access without writing custom integration code. Install friction is moderate (platform-specific wheels for Linux and macOS only), but maintenance is active and there are no known vulnerabilities. Verify whether you should install this directly or rely on it as a transitive dependency.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- AWS credentials required via EC2 Instance Role, AWS CLI, credential files (~/.aws/credentials), or environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY).
- PyTorch >= 2.0 required; PyTorch >= 2.3 for distributed checkpoint features.
- Pre-built wheels available for Linux and macOS only.
License · maintenance · safety
permissive license (permissive) — Permissive license treatment allows broad use without significant legal constraints.
last release 2026-02-20 (175 days)
0 known vulnerabilities (OSV.dev, 2026-08-14) · 1,405,841 downloads/mo, #3,944 on PyPI
Alternatives
Verify before relying
pip install s3torchconnectorclient
from s3torchconnectorclient import S3MapDataset
dataset = S3MapDataset.from_prefix("s3://bucket/prefix", region="us-east-1")
item = dataset[0]
print(item.key)- Whether this package is intended for direct installation or only as an internal dependency
- Specific performance improvements or throughput gains versus standard S3 access
- Full compatibility matrix with PyTorch versions outside the 2.0+ range
What it is and what it does
s3torchconnectorclient is an internal S3 client implementation that provides optimized access to Amazon S3 for PyTorch training workflows. It supports both map-style datasets for random access and iterable-style datasets for streaming sequential access, handling concurrent S3 requests and bucket listing automatically without requiring custom integration code.
The package includes checkpoint management interfaces for saving and loading model checkpoints directly to S3 without intermediate local storage. For distributed training, it implements PyTorch's StorageWriter and StorageReader interfaces. It supports Python 3.8 through 3.14 and requires PyTorch 2.0 or newer, with PyTorch 2.3 or newer needed for distributed checkpoint features.
Use it for
- Load training datasets from S3 buckets with automatic optimization for high-throughput data pipelines
- Save model checkpoints directly to S3 during training without writing to local disk first
- Implement distributed checkpointing across multiple training nodes using PyTorch's distributed checkpoint framework
- Stream sequential training data from S3 using iterable-style datasets for memory-efficient training
- Access S3 Express One Zone directory buckets for low-latency training data access
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you are training PyTorch models with data in Amazon S3 and want optimized, managed S3 access without writing custom integration code.
Install friction is moderate (platform-specific wheels for Linux and macOS only), but maintenance is active and there are no known vulnerabilities. Verify whether you should install this directly or rely on it as a transitive dependency.
Install
s3torchconnectorclient on PyPI
Before you install
Medium install friction due to platform-specific wheels for Linux and macOS only; macOS x86_64 support deprecated. Actively maintained with recent releases. No runtime dependencies to manage.
AWS credentials required via EC2 Instance Role, AWS CLI, credential files (~/.aws/credentials), or environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY). PyTorch >= 2.0 required; PyTorch >= 2.3 for distributed checkpoint features. Pre-built wheels available for Linux and macOS only.
License in practice
Permissive license treatment allows broad use without significant legal constraints.
Quickstart
pip install s3torchconnectorclient
from s3torchconnectorclient import S3MapDataset
dataset = S3MapDataset.from_prefix("s3://bucket/prefix", region="us-east-1")
item = dataset[0]
print(item.key)Verify before relying
- Whether this package is intended for direct installation or only as an internal dependency
- Specific performance improvements or throughput gains versus standard S3 access
- Full compatibility matrix with PyTorch versions outside the 2.0+ range
Package facts
| License | permissive license permissive |
| Python support | Supports the current Python release <3.15,>=3.8 |
| Install friction | Medium. Platform-specific wheel |
| Runtime dependencies | None |
| Maintenance | Actively maintained 175 days since the last release |
| First released | |
| Downloads | 1,405,841 / month, #3,944 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableLicense :: OSI Approved :: BSD LicenseOperating System :: OS IndependentProgramming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Programming Language :: Python :: 3.8Programming Language :: Python :: 3.9Topic :: Utilities |
Evidence: s3torchconnectorclient-1.5.0-cp310-cp310-macosx_10_12_x86_64.whl; s3torchconnectorclient-1.5.0-cp310-cp310-macosx_11_0_arm64.whl; s3torchconnectorclient-1.5.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; s3torchconnectorclient-1.5.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; s3torchconnectorclient-1.5.0-cp311-cp311-macosx_10_12_x86_64.whl; s3torchconnectorclient-1.5.0-cp311-cp311-macosx_11_0_arm64.whl; s3torchconnectorclient-1.5.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; s3torchconnectorclient-1.5.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; s3torchconnectorclient-1.5.0-cp312-cp312-macosx_10_13_x86_64.whl; s3torchconnectorclient-1.5.0-cp312-cp312-macosx_11_0_arm64.whl; s3torchconnectorclient-1.5.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; s3torchconnectorclient-1.5.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; s3torchconnectorclient-1.5.0-cp313-cp313-macosx_11_0_arm64.whl; s3torchconnectorclient-1.5.0-cp313-cp313-manylinux_2_28_aarch64.whl; s3torchconnectorclient-1.5.0-cp313-cp313-manylinux_2_28_x86_64.whl; s3torchconnectorclient-1.5.0-cp314-cp314-macosx_11_0_arm64.whl; s3torchconnectorclient-1.5.0-cp314-cp314-manylinux_2_28_aarch64.whl; s3torchconnectorclient-1.5.0-cp314-cp314-manylinux_2_28_x86_64.whl; s3torchconnectorclient-1.5.0-cp38-cp38-macosx_10_12_x86_64.whl; s3torchconnectorclient-1.5.0-cp38-cp38-macosx_11_0_arm64.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “amazon s3 torch connector”
- s3torchconnectorclientInternal S3 client implementation providing optimized data loading…
- PyAthenaPyAthena is a Python DB API 2.0 client for Amazon Athena that lets…
- apache-airflow-providers-amazonProvides Apache Airflow operators, hooks, and sensors for…
Give your agent the search over MCP, or paste the wish link into any chat.
More Utilities packages
Converts domain names between Unicode and ASCII-compatible encoding (Punycode) according to IDNA 2008 and Unicode Technical Standard 46, with security validation and broader script coverage than the standard library.
Install it if you work with internationalized domain names, need to validate domains, or use HTTP clients that depend on it transitively.
Detects and normalizes text encoding from unknown or ambiguous sources, supporting all IANA character sets that Python's core library provides codecs for, with the ability to register custom codecs.
Setuptools is a Python build backend and package management tool that handles building, distributing, and installing Python packages, including support for C/C++ extension modules.
Pluggy provides a plugin system that lets you define hook specifications and register implementations to be called in sequence, enabling extensible Python applications without tight coupling.
Install it if you're building an extensible application or framework.
Pygments is a syntax highlighter that colorizes source code and text in over 500 languages and formats, outputting to HTML, LaTeX, RTF, SVG, images, or ANSI terminal sequences.
Install it if you need to display or transform source code.
Six provides utility functions to write Python code that runs on both Python 2.7 and Python 3.3+, smoothing over language differences between the two versions.
See also s3torchconnector · litdata · synapse-s3-storage-provider · aws-cdk.aws-kinesisfirehose-alpha · megatron-fsdp · s3fs · mosaicml-streaming · torchtitan · tensorizer · aistore