{"categories":[{"label":"Utilities","url":"https://skillfed.io/packages/category/utilities/4"}],"enrichment":{"capability":"Provides PyTorch dataset primitives and checkpoint interfaces for efficient data loading and model checkpointing directly from Amazon S3, supporting both map-style and iterable-style datasets.","skillfed_tags":["pytorch-integration","aws-s3","distributed-training"],"use_cases":["Stream training data from S3 buckets into PyTorch DataLoaders without downloading to local disk first","Save model checkpoints directly to S3 during distributed training without staging through local storage","Load distributed checkpoints from S3 using optimized readers for faster recovery in multi-node training","Access S3 Express One Zone directory buckets for high-performance data access in time-sensitive training jobs","Implement random-access training datasets backed by S3 for flexible sampling patterns"],"what_it_does":"s3torchconnector bridges PyTorch's data loading pipeline with Amazon S3, eliminating the need to write custom code for bucket listing and concurrent request handling. It implements PyTorch's dataset primitives\u2014both map-style for random access and iterable-style for streaming\u2014so training jobs can fetch data directly from S3 with automatic performance optimization. It also provides checkpoint interfaces to save and load model state directly to S3 without intermediate local storage, and includes distributed checkpoint support via StorageWriter and StorageReader implementations compatible with PyTorch's distributed checkpoint framework.\n\nThe package depends on torch and s3torchconnectorclient (a compiled native component), which creates platform-specific installation requirements. It supports Python 3.8 through 3.14 and requires PyTorch 2.0 or newer (PyTorch 2.3+ for distributed checkpoint features). AWS credentials must be configured via EC2 instance role, AWS CLI, credential files, or environment variables before use.","worth_installing":"Yes, if you train PyTorch models on AWS and need to load data from or checkpoint to S3. High install friction (compiled deps, platform-specific wheels) and AWS credential configuration requirements are real costs, but the package is actively maintained, permissively licensed, and eliminates custom S3 integration code. Not worth installing if you're not on Linux/macOS or don't use S3 for training data."},"id":"s3torchconnector","links":{"html":"https://skillfed.io/packages/s3torchconnector","md":"https://skillfed.io/packages/s3torchconnector.md","pypi":"https://pypi.org/project/s3torchconnector/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-02-20","license_spdx":null,"license_treatment":"permissive","name":"s3torchconnector","python_support":"supports_current","summary":"S3 connector integration for PyTorch"},"popularity":{"monthly_downloads":1355211,"position":4008,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"1.5.0"}