snakemake-storage-plugin-s3

What it is and what it does

This package is a Snakemake storage plugin that extends Snakemake's workflow engine to treat S3-compatible object storage as a native file backend. Instead of managing local files or manually uploading to cloud storage, you can declare workflow outputs directly as S3 paths (e.g., s3://bucket/file.txt), and Snakemake will handle reading and writing through the S3 API. It works with AWS S3, MinIO, and other S3-compatible services.

The plugin integrates with Snakemake's plugin interface and relies on boto3 and botocore to communicate with S3 endpoints. Once installed, it becomes available to any Snakemake workflow on the same system. It is actively maintained and supports Python 3.11 through 3.13, with no known security vulnerabilities.

Use it for:

• Run Snakemake workflows that read input data from S3 buckets and write results back to S3 without intermediate local storage.
• Use MinIO or other S3-compatible storage as a workflow backend for on-premises or air-gapped environments.
• Integrate Snakemake pipelines with AWS data lakes where intermediate and final outputs live in S3.
• Simplify CI/CD pipelines by treating cloud object storage as a transparent file system within workflow definitions.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Enables Snakemake workflows to read from and write to S3-compatible storage (AWS S3, MinIO, and others) as a pluggable storage backend.

Yes, if you run Snakemake workflows that need S3 or S3-compatible storage integration. The plugin is actively maintained, has low install friction, carries a permissive MIT license, and has no known vulnerabilities. It is a straightforward way to enable cloud storage support in Snakemake without custom scripting.

Install

snakemake-storage-plugin-s3 on PyPI

pip install snakemake-storage-plugin-s3

uv add snakemake-storage-plugin-s3

poetry add snakemake-storage-plugin-s3

Installing snakemake-storage-plugin-s3

Before you install

Low friction install with a pure-Python wheel. Actively maintained as of April 2026, with recent updates. Depends on boto3 and botocore for AWS SDK functionality, which are standard and well-maintained.

License in practice

MIT license permits commercial and private use with minimal restrictions—suitable for most projects without licensing concerns.

Quickstart

pip install snakemake-storage-plugin-s3

# In a Snakefile, reference S3 paths directly:
# rule example:
#     output: "s3://my-bucket/output.txt"
#     shell: "echo hello > {output}"

Requires Python 3.11 or later. AWS credentials or MinIO endpoint configuration must be available in the environment (via AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, or similar).

Verify before relying

• Whether the plugin auto-discovers S3 credentials from IAM roles, environment variables, or config files, or requires explicit setup.
• Performance characteristics and any limits on object size or concurrent operations.
• Whether MinIO and other S3-compatible endpoints are tested and fully supported or require additional configuration.

Package facts

Evidence: snakemake_storage_plugin_s3-0.3.6-py3-none-any.whl

Keywords: snakemake, plugin, storage, s3

Tags

More Build Tools packages