icechunk

What it is and what it does

Icechunk augments Zarr with transactional guarantees and version control for multidimensional array data stored on cloud object storage. It presents a key-value interface to Zarr while managing all I/O, enabling safe parallel reads and writes from multiple uncoordinated processes without external locks or databases. A repo is a self-contained Zarr store (typically a group with multiple related arrays) that supports snapshots, branches, tags, and time-travel access to previous versions.

The package is built on Rust bindings (PyO3) and provides serializable isolation—reads always see committed snapshots, writes are atomic and never partially visible. It supports chunk sharding, chunk references to external formats like HDF5, and schema evolution. Icechunk is designed for scientific computing, data science, and AI/ML workflows where consistent, versioned tensor storage on cloud infrastructure is required.

Use it for:

• Store and version large multidimensional datasets (e.g., climate, geospatial, or scientific simulations) on S3 or other cloud storage with transactional safety.
• Enable multiple concurrent processes to read and write to the same Zarr arrays without coordination or locking.
• Maintain immutable snapshots and branches of tensor data for reproducibility and time-travel debugging.
• Organize related arrays into hierarchical groups with schema evolution (add, rename, remove arrays with minimal overhead).
• Reference chunks from external file formats (HDF5, NetCDF) within a Zarr-compatible store.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Icechunk is a transactional storage engine for tensor/ND-array data that works with Zarr to provide consistent, multi-process-safe reads and writes on cloud object storage.

Yes, if you need transactional, multi-process-safe tensor storage on cloud object storage with Zarr. The package is actively maintained, has no known vulnerabilities, and supports current Python versions. However, verify the license treatment before use in proprietary contexts, and confirm that the Apache 2.0 license mentioned in the description is formally declared. Medium install friction is acceptable for most environments given the availability of prebuilt wheels.

Install

icechunk on PyPI

pip install icechunk

uv add icechunk

poetry add icechunk

Installing icechunk

Before you install

Medium install friction due to compiled Rust bindings; wheels are available for current Python versions (3.12+) and common platforms (x86_64, arm64, macOS, Linux, Windows). Active maintenance with recent release.

License in practice

License treatment is unclear; the description excerpt mentions Apache 2.0, but the metadata does not confirm this. Verify the actual license before use in proprietary or restricted contexts.

Quickstart

pip install icechunk

import zarr
from icechunk import IcechunkStore

store = IcechunkStore(root="s3://bucket/path")
group = zarr.open_group(store=store, mode="r+")

Requires Python 3.12 or later; Zarr must be installed as a runtime dependency.

Verify before relying

• Whether Apache 2.0 license claimed in description excerpt is formally declared in package metadata
• Specific cloud storage backends supported beyond the description's mention of GCS, Azure, and S3
• Performance characteristics and scalability limits for concurrent transactions
• Compatibility with Zarr implementations other than Zarr Python

Package facts

Evidence: icechunk-2.1.2-cp312-abi3-macosx_10_12_x86_64.whl; icechunk-2.1.2-cp312-abi3-macosx_11_0_arm64.whl; icechunk-2.1.2-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; icechunk-2.1.2-cp312-abi3-manylinux_2_28_aarch64.whl; icechunk-2.1.2-cp312-abi3-musllinux_1_2_aarch64.whl; icechunk-2.1.2-cp312-abi3-musllinux_1_2_x86_64.whl; icechunk-2.1.2-cp312-abi3-win_amd64.whl

Tags

More Database packages