---
id: array-record
version: "0.8.3"
license: Apache-2.0
license_treatment: permissive
maintenance: active
---

# array-record — A file format that achieves a new frontier of IO efficiency

License: permissive · Maintenance: active · Downloads: 3.6M/mo

## What it is and what it does

ArrayRecord is a file format designed for efficient storage and retrieval of array data, derived from Google's Riegeli format. It supports parallel read and write operations, random access by record index, and reuses Riegeli's compression algorithms. The package provides Python bindings to work with ArrayRecord files, making it suitable for machine learning pipelines, data processing workflows, and other scenarios where you need fast, indexed access to serialized array data.

The package depends on absl-py and etils, and requires Python 3.11 or later. It ships as precompiled wheels for modern Python versions across macOS and Linux platforms, reducing installation complexity. The format is particularly relevant for systems that need to read or write large collections of records in parallel without sequential bottlenecks.

Use it for:

- Store and retrieve training datasets for machine learning with parallel I/O in data pipelines
- Random-access indexing into large serialized record collections without sequential scanning
- Replace Riegeli when you need parallel read/write and indexed record lookup in Python
- Efficient data interchange format for distributed computing frameworks handling array data

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

ArrayRecord is a file format for storing and accessing array data with parallel read/write support and random access by record index, built on Riegeli compression.

Yes, if you need parallel record I/O and random indexing for array data. The active maintenance, Apache-2.0 license, and prebuilt wheels make it low-friction to adopt. Install it when your workflow involves large record collections that benefit from indexed access and parallel operations; skip it if you only need sequential streaming or don't require index-based lookups.

## Install

pip install array-record
uv add array-record
poetry add array-record

## Installing array-record

Before you install: Medium install friction due to compiled wheels for multiple Python versions and architectures. Active maintenance with recent commits; last release 274 days ago suggests ongoing development.

License in practice: Apache-2.0 permissive license allows commercial and private use with minimal restrictions; suitable for most projects.

Quickstart:

pip install array-record

import array_record
# Create or read ArrayRecord files with parallel I/O support

Requires Python 3.11 or later; prebuilt wheels available for macOS (arm64) and Linux (aarch64, x86_64).

Verify before relying:

- Specific API surface and how to instantiate/use ArrayRecord readers and writers
- Performance benchmarks vs. Riegeli or other serialization formats
- Whether random access by index requires loading entire file into memory

## Package facts

- License: Apache-2.0 (permissive)
- Python support: supports_current
- Install friction: medium
- Maintenance: active
- Downloads: 3.6M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

efficient array file format, parallel record read write, random access record storage, riegeli-based file format, high-performance data serialization, data-serialization, parallel-io, array-storage

[View on SkillFed](https://skillfed.io/packages/array-record) · [View on PyPI](https://pypi.org/project/array-record/)