obstore

What it is and what it does

obstore is a Rust-backed Python library that abstracts away the differences between cloud storage providers—S3, Google Cloud Storage, Azure Storage, and S3-compatible services—behind a single, type-hinted interface. It handles both synchronous and asynchronous operations, streaming data transfer, automatic multipart uploads for large files, and credential refresh, all with minimal Python dependencies (only typing-extensions at runtime).

The package is designed for high-throughput workloads where you need to interact with object storage from Python without the overhead of traditional clients. It exposes file-like APIs, integrates with fsspec, and can return list results as Apache Arrow tables for memory efficiency. Zero-copy data exchange via the buffer protocol minimizes overhead between Rust and Python.

Use it for:

• Bulk download or upload of files from multiple cloud providers using a single API without provider-specific code.
• High-concurrency async workloads that need to fetch many small objects from S3 or compatible storage efficiently.
• Streaming large files from cloud storage to avoid loading entire objects into memory.
• Automated credential refresh in long-running applications that interact with expiring cloud credentials.
• Integration with geospatial or data science pipelines that read from cloud-hosted datasets via fsspec.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

obstore provides a unified Python interface to Amazon S3, Google Cloud Storage, Azure Storage, and S3-compliant APIs, with both sync and async support, streaming uploads and downloads, and automatic multipart handling.

Yes, if you need a unified, high-performance interface to multiple cloud storage backends. The permissive MIT license, active maintenance, lack of heavy Python dependencies, and strong async support make it a solid choice for cloud-native Python applications. Medium install friction (compiled wheels) is manageable on common platforms. No known vulnerabilities.

Install

obstore on PyPI

pip install obstore

uv add obstore

poetry add obstore

Installing obstore

Before you install

Medium install friction due to compiled Rust bindings, but wheels are available for many platforms including macOS, Linux, and Windows across multiple architectures. Active maintenance with a recent release.

License in practice

Licensed under MIT (permissive), so you can use it freely in commercial and private projects without restriction.

Quickstart

pip install obstore

import obstore

# Create S3 client and get object
store = obstore.S3Store()
data = store.get("s3://bucket/key")

Requires Python 3.10 or later. Compiled wheels available for most platforms, but installation may require a build environment on unsupported architectures.

Verify before relying

• Whether the 9x throughput claim vs fsspec and 2.8x vs aioboto3 holds across your specific workload and concurrency patterns.
• Whether Apache Arrow format support requires additional dependencies or configuration beyond what's listed.

Package facts

Evidence: obstore-0.11.0-cp310-cp310-macosx_10_12_x86_64.whl; obstore-0.11.0-cp310-cp310-macosx_11_0_arm64.whl; obstore-0.11.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl; obstore-0.11.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl; obstore-0.11.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl; obstore-0.11.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl; obstore-0.11.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; obstore-0.11.0-cp310-cp310-manylinux_2_24_aarch64.whl; obstore-0.11.0-cp310-cp310-musllinux_1_2_aarch64.whl; obstore-0.11.0-cp310-cp310-musllinux_1_2_armv7l.whl; obstore-0.11.0-cp310-cp310-musllinux_1_2_i686.whl; obstore-0.11.0-cp310-cp310-musllinux_1_2_x86_64.whl; obstore-0.11.0-cp310-cp310-win_amd64.whl; obstore-0.11.0-cp311-abi3-macosx_10_12_x86_64.whl; obstore-0.11.0-cp311-abi3-macosx_11_0_arm64.whl; obstore-0.11.0-cp311-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl; obstore-0.11.0-cp311-abi3-manylinux_2_17_i686.manylinux2014_i686.whl; obstore-0.11.0-cp311-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl; obstore-0.11.0-cp311-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl; obstore-0.11.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Tags

More Internet packages