kerchunk

What it is and what it does

Kerchunk is a metadata extraction library that transforms archival data formats into cloud-friendly virtual datasets. Instead of copying or converting NetCDF, HDF5, GRIB, TIFF, FITS, or Zarr files, it extracts byte ranges, compression metadata, and structural information, storing this as a separate reference object. This allows you to create unified, queryable datasets spanning many source files without moving the original data.

The library integrates with fsspec to read from diverse storage backends—S3, GCS, HTTP, local filesystems, and network protocols—and with zarr for parallel, lock-free access. It supports asynchronous concurrent fetching of data chunks and coordinate-based indexing across arbitrary dimensions, making it a gateway for serverless, in-situ processing of massive archival datasets in the cloud while data providers continue using legacy formats.

Use it for:

• Aggregate NetCDF climate or weather data from thousands of files into a single queryable dataset without copying.
• Access HDF5 scientific data stored in cloud object storage (S3, GCS) with efficient byte-range requests.
• Create virtual GRIB datasets for meteorological analysis spanning multiple time steps and sources.
• Build logical views over heterogeneous file types (mix of NetCDF and HDF5) with unified coordinate indexing.
• Enable serverless data processing pipelines that fetch only required chunks from archival storage.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Kerchunk extracts metadata from chunked, compressed data formats (NetCDF, HDF5, GRIB, TIFF, FITS, Zarr) and stores it separately, enabling efficient cloud-friendly access to archival data without copying or translating original files.

Yes. Kerchunk solves a specific, high-value problem for scientific and climate data workflows: efficient cloud access to legacy archival formats without data duplication. Low install friction, active maintenance, permissive licensing, no known vulnerabilities, and a focused dependency set make it a safe choice for teams working with large multi-file datasets in cloud environments.

Install

kerchunk on PyPI

pip install kerchunk

uv add kerchunk

poetry add kerchunk

Installing kerchunk

Before you install

Low install friction with a pure-Python wheel. Active maintenance as of 2026-03-30 with 367 repository stars. Requires Python 3.11 or later. Five runtime dependencies (fsspec, numcodecs, numpy, ujson, zarr) are all widely used data-science libraries.

License in practice

MIT license (permissive) means you can use, modify, and distribute kerchunk freely in commercial and private projects with minimal restrictions, provided you include the license notice.

Quickstart

pip install kerchunk

import kerchunk.hdf
from fsspec.implementations.reference import ReferenceFileSystem

# Extract metadata from HDF5 file
reference_dict = kerchunk.hdf.SingleHdf5ToZarr('data.h5').translate()

# Access via virtual dataset
fs = ReferenceFileSystem(reference_dict)
data = fs.open('data', 'rb')

Requires Python 3.11 or later. Source data files must be in a supported format (NetCDF, HDF5, GRIB, TIFF, FITS, or Zarr) and accessible via fsspec-supported storage backends.

Verify before relying

• Specific performance gains or latency amortization metrics for concurrent chunk fetching compared to direct file access.
• Scalability limits for datasets with millions of files and typical query response times.
• Compatibility matrix for heterogeneous file types within a single virtual dataset.
• Memory overhead of metadata consolidation for large multi-file datasets.

Package facts

Evidence: kerchunk-0.2.10-py3-none-any.whl

Tags

More Scientific/Engineering packages