array-record
A file format that achieves a new frontier of IO efficiency
Decision gist · record as of 2026-08-14
Yes, if you need parallel record I/O and random indexing for array data. The active maintenance, Apache-2.0 license, and prebuilt wheels make it low-friction to adopt. Install it when your workflow involves large record collections that benefit from indexed access and parallel operations; skip it if you only need sequential streaming or don't require index-based lookups.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.11 or later; prebuilt wheels available for macOS (arm64) and Linux (aarch64, x86_64).
- Medium install friction due to compiled wheels for multiple Python versions and architectures.
- Active maintenance with recent commits; last release 274 days ago suggests ongoing development.
License · maintenance · safety
Apache-2.0 (permissive) — Apache-2.0 permissive license allows commercial and private use with minimal restrictions; suitable for most projects.
last release 2025-11-13 (274 days) · last repo commit 2026-08-13 · 139 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 3,636,122 downloads/mo, #2,548 on PyPI
Alternatives
Verify before relying
pip install array-record
import array_record
# Create or read ArrayRecord files with parallel I/O support- Specific API surface and how to instantiate/use ArrayRecord readers and writers
- Performance benchmarks vs. Riegeli or other serialization formats
- Whether random access by index requires loading entire file into memory
What it is and what it does
ArrayRecord is a file format designed for efficient storage and retrieval of array data, derived from Google's Riegeli format. It supports parallel read and write operations, random access by record index, and reuses Riegeli's compression algorithms. The package provides Python bindings to work with ArrayRecord files, making it suitable for machine learning pipelines, data processing workflows, and other scenarios where you need fast, indexed access to serialized array data.
The package depends on absl-py and etils, and requires Python 3.11 or later. It ships as precompiled wheels for modern Python versions across macOS and Linux platforms, reducing installation complexity. The format is particularly relevant for systems that need to read or write large collections of records in parallel without sequential bottlenecks.
Use it for
- Store and retrieve training datasets for machine learning with parallel I/O in data pipelines
- Random-access indexing into large serialized record collections without sequential scanning
- Replace Riegeli when you need parallel read/write and indexed record lookup in Python
- Efficient data interchange format for distributed computing frameworks handling array data
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you need parallel record I/O and random indexing for array data.
The active maintenance, Apache-2.0 license, and prebuilt wheels make it low-friction to adopt. Install it when your workflow involves large record collections that benefit from indexed access and parallel operations; skip it if you only need sequential streaming or don't require index-based lookups.
Install
array-record on PyPI
Before you install
Medium install friction due to compiled wheels for multiple Python versions and architectures. Active maintenance with recent commits; last release 274 days ago suggests ongoing development.
Requires Python 3.11 or later; prebuilt wheels available for macOS (arm64) and Linux (aarch64, x86_64).
License in practice
Apache-2.0 permissive license allows commercial and private use with minimal restrictions; suitable for most projects.
Quickstart
pip install array-record
import array_record
# Create or read ArrayRecord files with parallel I/O supportVerify before relying
- Specific API surface and how to instantiate/use ArrayRecord readers and writers
- Performance benchmarks vs. Riegeli or other serialization formats
- Whether random access by index requires loading entire file into memory
Package facts
| License | Apache-2.0 permissive |
| Python support | Supports the current Python release >=3.11 |
| Install friction | Medium. Platform-specific wheel |
| Runtime dependencies | 2 packagesabsl-pyetils |
| Maintenance | Actively maintained 274 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 3,636,122 / month, #2,548 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14 |
Evidence: array_record-0.8.3-cp311-cp311-macosx_11_0_arm64.whl; array_record-0.8.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; array_record-0.8.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; array_record-0.8.3-cp312-cp312-macosx_11_0_arm64.whl; array_record-0.8.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; array_record-0.8.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; array_record-0.8.3-cp313-cp313-macosx_11_0_arm64.whl; array_record-0.8.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; array_record-0.8.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; array_record-0.8.3-cp314-cp314-macosx_11_0_arm64.whl; array_record-0.8.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; array_record-0.8.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “efficient array file format”
- array-recordArrayRecord is a file format for storing and accessing array data…
- asdfASDF is a Python implementation of the Advanced Scientific Data…
- vortex-dataVortex-data provides Python bindings to work with Vortex, a columnar…
Give your agent the search over MCP, or paste the wish link into any chat.
More Database packages
psycopg2-binary is a PostgreSQL database adapter for Python that implements the DB API 2.0 specification, enabling Python applications to connect to and query PostgreSQL databases with thread-safe concurrent operations.
Python client library for connecting to and executing commands against Redis key-value stores, supporting both synchronous and asynchronous operations.
Install it if your application needs to interact with Redis; the only prerequisite is a running Redis server instance.
YDB Python SDK is the official client library for connecting to and querying YDB databases from Python applications.
Install it if you need to connect Python applications to YDB databases.
Connects Python applications to Snowflake data warehouses using the DB API 2.0 specification, enabling SQL queries, data transfers, and warehouse operations.
sqlparse tokenizes SQL text into a tree of statements, clauses, and expressions, and provides functions to split scripts, format queries, and inspect parsed tokens without validating dialect or syntax.
Install it if you need to manipulate, format, or analyze SQL text programmatically.
Provides base adapter protocols and shared functionality that database adapters use to integrate with dbt-core, handling connections, dialect translation, relation caching, and core interface management.
See also acvl-utils · rensa · awkward-cpp · awkward0 · numexpr · HLL · awkward · libkvikio-cu12 · tensorstore