blobfile

What it is and what it does

Blobfile abstracts away the differences between local filesystem operations and cloud blob storage by exposing a single Python interface modeled on the built-in `open()` function. You can read and write files on Google Cloud Storage (gs://), Azure Blob Storage (az:// or https://), and local paths using the same code, with optional streaming to avoid downloading entire files into memory.

The library provides both high-level file operations (BlobFile for opening, copy for remote-to-remote transfers, exists, listdir, walk, stat, remove, rmtree) and path utilities (basename, dirname, join) that work uniformly across all three backends. It handles authentication through environment variables or application default credentials, and offers configuration options for connection pooling, timeouts, chunk sizes, and caching behavior.

Use it for:

• Process large datasets stored in GCS or Azure without downloading them entirely to local disk first
• Write data processing pipelines that work with both local development files and cloud-hosted production data
• Migrate file operations between cloud providers by changing only the path prefix, not the code
• Implement parallel blob processing with built-in support for concurrent operations across cloud storage
• Cache remote files locally for repeated reads while maintaining a unified file interface

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides a unified Python file interface for reading and writing to local filesystems, Google Cloud Storage, and Azure Blob Storage using an API similar to the built-in `open()` function.

Yes, if you work with cloud blob storage and want a simpler, provider-agnostic file interface. The low install friction, active maintenance, public domain license, and lack of known vulnerabilities make it a safe choice. Install only if you actually need multi-cloud or local-plus-cloud file abstraction; if you use only one provider, that provider's native SDK may be more direct.

Install

blobfile on PyPI

pip install blobfile

uv add blobfile

poetry add blobfile

Installing blobfile

Before you install

Low friction: pure Python wheel with four straightforward runtime dependencies (pycryptodomex, urllib3, lxml, filelock). Repository is active with recent commits and no known vulnerabilities.

License in practice

Public Domain license means no restrictions on use, modification, or distribution—you can use this freely in any project without license compatibility concerns.

Quickstart

pip install blobfile

import blobfile as bf

with bf.BlobFile("gs://my-bucket-name/cats", "wb") as f:
    f.write(b"meow!")

with bf.BlobFile("gs://my-bucket-name/cats", "rb") as f:
    print(f.read())

Requires Python 3.10 or later. Authentication to GCS or Azure requires appropriate credentials (environment variables or default application credentials).

Verify before relying

• Performance characteristics and latency overhead compared to cloud provider SDKs
• Concurrent access limits or connection pooling behavior under high load
• Compatibility with specific GCS/Azure API versions or deprecated endpoints

Package facts

Evidence: blobfile-3.2.0-py3-none-any.whl

Tags

More Utilities packages