---
id: kerchunk
version: "0.2.10"
license: MIT
license_treatment: permissive
maintenance: active
---

# kerchunk — Functions to make reference descriptions for ReferenceFileSystem

License: permissive · Maintenance: active · Downloads: 138.0K/mo

## What it is and what it does

Kerchunk is a metadata extraction library that transforms archival data formats into cloud-friendly virtual datasets. Instead of copying or converting NetCDF, HDF5, GRIB, TIFF, FITS, or Zarr files, it extracts byte ranges, compression metadata, and structural information, storing this as a separate reference object. This allows you to create unified, queryable datasets spanning many source files without moving the original data.

The library integrates with fsspec to read from diverse storage backends—S3, GCS, HTTP, local filesystems, and network protocols—and with zarr for parallel, lock-free access. It supports asynchronous concurrent fetching of data chunks and coordinate-based indexing across arbitrary dimensions, making it a gateway for serverless, in-situ processing of massive archival datasets in the cloud while data providers continue using legacy formats.

Use it for:

- Aggregate NetCDF climate or weather data from thousands of files into a single queryable dataset without copying.
- Access HDF5 scientific data stored in cloud object storage (S3, GCS) with efficient byte-range requests.
- Create virtual GRIB datasets for meteorological analysis spanning multiple time steps and sources.
- Build logical views over heterogeneous file types (mix of NetCDF and HDF5) with unified coordinate indexing.
- Enable serverless data processing pipelines that fetch only required chunks from archival storage.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Kerchunk extracts metadata from chunked, compressed data formats (NetCDF, HDF5, GRIB, TIFF, FITS, Zarr) and stores it separately, enabling efficient cloud-friendly access to archival data without copying or translating original files.

Yes. Kerchunk solves a specific, high-value problem for scientific and climate data workflows: efficient cloud access to legacy archival formats without data duplication. Low install friction, active maintenance, permissive licensing, no known vulnerabilities, and a focused dependency set make it a safe choice for teams working with large multi-file datasets in cloud environments.

## Install

pip install kerchunk
uv add kerchunk
poetry add kerchunk

## Installing kerchunk

Before you install: Low install friction with a pure-Python wheel. Active maintenance as of 2026-03-30 with 367 repository stars. Requires Python 3.11 or later. Five runtime dependencies (fsspec, numcodecs, numpy, ujson, zarr) are all widely used data-science libraries.

License in practice: MIT license (permissive) means you can use, modify, and distribute kerchunk freely in commercial and private projects with minimal restrictions, provided you include the license notice.

Quickstart:

pip install kerchunk

import kerchunk.hdf
from fsspec.implementations.reference import ReferenceFileSystem

# Extract metadata from HDF5 file
reference_dict = kerchunk.hdf.SingleHdf5ToZarr('data.h5').translate()

# Access via virtual dataset
fs = ReferenceFileSystem(reference_dict)
data = fs.open('data', 'rb')

Requires Python 3.11 or later. Source data files must be in a supported format (NetCDF, HDF5, GRIB, TIFF, FITS, or Zarr) and accessible via fsspec-supported storage backends.

Verify before relying:

- Specific performance gains or latency amortization metrics for concurrent chunk fetching compared to direct file access.
- Scalability limits for datasets with millions of files and typical query response times.
- Compatibility matrix for heterogeneous file types within a single virtual dataset.
- Memory overhead of metadata consolidation for large multi-file datasets.

## Package facts

- License: MIT (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 138.0K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

cloud-friendly data access, reference file system metadata, virtual datasets from multiple files, chunked data format access, in-situ cloud data processing, metadata extraction for archival formats, serverless data aggregation, cloud-storage, scientific-data, metadata-extraction

[View on SkillFed](https://skillfed.io/packages/kerchunk) · [View on PyPI](https://pypi.org/project/kerchunk/)