tensorflow-io-gcs-filesystem

What it is and what it does

tensorflow-io-gcs-filesystem is a TensorFlow I/O extension that adds Google Cloud Storage filesystem support to TensorFlow. It allows you to read and write data directly from GCS buckets within TensorFlow data pipelines, eliminating the need to download datasets locally before processing. The package integrates with tf.data.Dataset and Keras workflows to stream data from cloud storage.

The package is distributed as platform-specific compiled wheels for Python 3.7–3.12 on macOS (x86_64 and ARM64) and Linux (x86_64 and aarch64). It has no runtime dependencies beyond TensorFlow itself and is actively maintained, with the latest release (0.37.1) compatible with TensorFlow 2.16.x. Installation requires matching your TensorFlow version to the compatibility table provided in the documentation.

Use it for:

• Load training datasets directly from GCS URLs in machine learning workflows without downloading to disk first.
• Stream large datasets from cloud storage in data pipelines for memory-efficient processing.
• Access public datasets hosted on GCS via HTTP/HTTPS URLs in TensorFlow code.
• Build data preprocessing workflows that read raw data from GCS and write processed results back to cloud storage.
• Integrate GCS file access into distributed TensorFlow training on Google Cloud Platform infrastructure.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides Google Cloud Storage filesystem support for TensorFlow, enabling direct reading and writing of data from GCS buckets within TensorFlow pipelines without local downloads.

Yes, if you are building TensorFlow pipelines that consume data from Google Cloud Storage and want to avoid local downloads. Install only if your TensorFlow version matches the compatibility table (0.37.1 requires TensorFlow 2.16.x). No known vulnerabilities. Medium install friction due to compiled wheels, but straightforward once the correct TensorFlow version is in place.

Install

tensorflow-io-gcs-filesystem on PyPI

pip install tensorflow-io-gcs-filesystem

uv add tensorflow-io-gcs-filesystem

poetry add tensorflow-io-gcs-filesystem

Installing tensorflow-io-gcs-filesystem

Before you install

Medium install friction due to platform-specific compiled wheels (cp39–cp312 across macOS and Linux architectures). Actively maintained with recent commits; compatible with TensorFlow 2.16.x per version table.

License in practice

Licensed under Apache Software License (permissive), allowing commercial and private use with minimal restrictions.

Quickstart

pip install tensorflow-io-gcs-filesystem

# Read dataset directly from GCS URLs
dataset_url = "https://storage.googleapis.com/cvdf-datasets/mnist/"
d_train = tfio.IODataset.from_mnist(
    dataset_url + "train-images-idx3-ubyte.gz",
    dataset_url + "train-labels-idx1-ubyte.gz"
)
d_train = d_train.batch(32)

Requires TensorFlow 2.16.x installed; Python 3.7–3.12 supported; GCS credentials must be configured in your environment.

Verify before relying

• Whether this package is a standalone filesystem plugin or requires the full tensorflow-io package as a peer dependency.
• Exact GCS authentication method expected (Application Default Credentials, service account key, etc.).
• Performance characteristics when reading large datasets from GCS vs. local storage.
• Whether automatic decompression of gzipped files is supported by this filesystem component.

Package facts

Evidence: tensorflow_io_gcs_filesystem-0.37.1-cp310-cp310-macosx_10_14_x86_64.whl; tensorflow_io_gcs_filesystem-0.37.1-cp310-cp310-macosx_12_0_arm64.whl; tensorflow_io_gcs_filesystem-0.37.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; tensorflow_io_gcs_filesystem-0.37.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; tensorflow_io_gcs_filesystem-0.37.1-cp311-cp311-macosx_10_14_x86_64.whl; tensorflow_io_gcs_filesystem-0.37.1-cp311-cp311-macosx_12_0_arm64.whl; tensorflow_io_gcs_filesystem-0.37.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; tensorflow_io_gcs_filesystem-0.37.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; tensorflow_io_gcs_filesystem-0.37.1-cp312-cp312-macosx_10_14_x86_64.whl; tensorflow_io_gcs_filesystem-0.37.1-cp312-cp312-macosx_12_0_arm64.whl; tensorflow_io_gcs_filesystem-0.37.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; tensorflow_io_gcs_filesystem-0.37.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; tensorflow_io_gcs_filesystem-0.37.1-cp39-cp39-macosx_10_14_x86_64.whl; tensorflow_io_gcs_filesystem-0.37.1-cp39-cp39-macosx_12_0_arm64.whl; tensorflow_io_gcs_filesystem-0.37.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; tensorflow_io_gcs_filesystem-0.37.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Keywords: tensorflow, io, machine, learning

Tags

More Software Development packages