array-record

What it is and what it does

ArrayRecord is a file format designed for efficient storage and retrieval of array data, derived from Google's Riegeli format. It supports parallel read and write operations, random access by record index, and reuses Riegeli's compression algorithms. The package provides Python bindings to work with ArrayRecord files, making it suitable for machine learning pipelines, data processing workflows, and other scenarios where you need fast, indexed access to serialized array data.

The package depends on absl-py and etils, and requires Python 3.11 or later. It ships as precompiled wheels for modern Python versions across macOS and Linux platforms, reducing installation complexity. The format is particularly relevant for systems that need to read or write large collections of records in parallel without sequential bottlenecks.

Use it for:

• Store and retrieve training datasets for machine learning with parallel I/O in data pipelines
• Random-access indexing into large serialized record collections without sequential scanning
• Replace Riegeli when you need parallel read/write and indexed record lookup in Python
• Efficient data interchange format for distributed computing frameworks handling array data

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

ArrayRecord is a file format for storing and accessing array data with parallel read/write support and random access by record index, built on Riegeli compression.

Yes, if you need parallel record I/O and random indexing for array data. The active maintenance, Apache-2.0 license, and prebuilt wheels make it low-friction to adopt. Install it when your workflow involves large record collections that benefit from indexed access and parallel operations; skip it if you only need sequential streaming or don't require index-based lookups.

Install

array-record on PyPI

pip install array-record

uv add array-record

poetry add array-record

Installing array-record

Before you install

Medium install friction due to compiled wheels for multiple Python versions and architectures. Active maintenance with recent commits; last release 274 days ago suggests ongoing development.

License in practice

Apache-2.0 permissive license allows commercial and private use with minimal restrictions; suitable for most projects.

Quickstart

pip install array-record

import array_record
# Create or read ArrayRecord files with parallel I/O support

Requires Python 3.11 or later; prebuilt wheels available for macOS (arm64) and Linux (aarch64, x86_64).

Verify before relying

• Specific API surface and how to instantiate/use ArrayRecord readers and writers
• Performance benchmarks vs. Riegeli or other serialization formats
• Whether random access by index requires loading entire file into memory

Package facts

Evidence: array_record-0.8.3-cp311-cp311-macosx_11_0_arm64.whl; array_record-0.8.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; array_record-0.8.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; array_record-0.8.3-cp312-cp312-macosx_11_0_arm64.whl; array_record-0.8.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; array_record-0.8.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; array_record-0.8.3-cp313-cp313-macosx_11_0_arm64.whl; array_record-0.8.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; array_record-0.8.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; array_record-0.8.3-cp314-cp314-macosx_11_0_arm64.whl; array_record-0.8.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; array_record-0.8.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Tags

More Database packages