{"categories":[{"label":"Scientific/Engineering","url":"https://skillfed.io/packages/category/scientific-engineering/7"}],"enrichment":{"capability":"Kerchunk extracts metadata from chunked, compressed data formats (NetCDF, HDF5, GRIB, TIFF, FITS, Zarr) and stores it separately, enabling efficient cloud-friendly access to archival data without copying or translating original files.","skillfed_tags":["cloud-storage","scientific-data","metadata-extraction"],"use_cases":["Aggregate NetCDF climate or weather data from thousands of files into a single queryable dataset without copying.","Access HDF5 scientific data stored in cloud object storage (S3, GCS) with efficient byte-range requests.","Create virtual GRIB datasets for meteorological analysis spanning multiple time steps and sources.","Build logical views over heterogeneous file types (mix of NetCDF and HDF5) with unified coordinate indexing.","Enable serverless data processing pipelines that fetch only required chunks from archival storage."],"what_it_does":"Kerchunk is a metadata extraction library that transforms archival data formats into cloud-friendly virtual datasets. Instead of copying or converting NetCDF, HDF5, GRIB, TIFF, FITS, or Zarr files, it extracts byte ranges, compression metadata, and structural information, storing this as a separate reference object. This allows you to create unified, queryable datasets spanning many source files without moving the original data.\n\nThe library integrates with fsspec to read from diverse storage backends\u2014S3, GCS, HTTP, local filesystems, and network protocols\u2014and with zarr for parallel, lock-free access. It supports asynchronous concurrent fetching of data chunks and coordinate-based indexing across arbitrary dimensions, making it a gateway for serverless, in-situ processing of massive archival datasets in the cloud while data providers continue using legacy formats.","worth_installing":"Yes. Kerchunk solves a specific, high-value problem for scientific and climate data workflows: efficient cloud access to legacy archival formats without data duplication. Low install friction, active maintenance, permissive licensing, no known vulnerabilities, and a focused dependency set make it a safe choice for teams working with large multi-file datasets in cloud environments."},"id":"kerchunk","links":{"html":"https://skillfed.io/packages/kerchunk","md":"https://skillfed.io/packages/kerchunk.md","pypi":"https://pypi.org/project/kerchunk/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-03-30","license_spdx":null,"license_treatment":"permissive","name":"kerchunk","python_support":"supports_current","summary":"Functions to make reference descriptions for ReferenceFileSystem"},"popularity":{"monthly_downloads":137953,"position":11348,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.2.10"}