iden

What it is and what it does

iden is a Python library for organizing and accessing machine learning training data split into shards—discrete data chunks that can be stored in different formats and loaded on demand. It abstracts away the mechanics of managing train/validation/test splits, persisting shards to disk, and retrieving them lazily so you don't load everything into memory at once. Each shard has a URI for reproducible identification and optional caching for frequently accessed data.

The library depends on coola (for data comparison) and objectory (for dynamic object instantiation), and supports formats like JSON, YAML, Pickle, PyTorch tensors, and safetensors. It's designed for the common ML workflow where you organize data into logical splits and want to load individual shards on demand rather than materializing the entire dataset upfront.

Use it for:

• Organize large training datasets into splits (train/val/test) and load shards lazily during model training.
• Store preprocessed data in multiple formats and switch between them without rewriting shard management code.
• Cache frequently accessed shards in memory while keeping the full dataset on disk to manage memory constraints.
• Persist dataset structure and shard references using URIs for reproducible data pipelines across runs.
• Build custom shard loaders for domain-specific data formats by extending the library's extensible architecture.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

iden manages machine learning datasets organized into shards with lazy loading, supporting multiple storage formats (JSON, YAML, Pickle, PyTorch, safetensors) and optional in-memory caching.

Yes. iden is actively maintained, has no known vulnerabilities, installs with minimal friction, and solves a concrete problem in ML workflows—organizing and lazily loading sharded datasets. The permissive BSD-3-Clause license poses no restriction. The API is pre-1.0 and may change, so pin the version if stability is critical, but for new projects or exploratory work it is a solid choice.

Install

iden on PyPI

pip install iden

uv add iden

poetry add iden

Installing iden

Before you install

Low friction: pure Python wheel with only two runtime dependencies (coola and objectory). Actively maintained with a recent release 60 days ago and current commit activity.

License in practice

BSD-3-Clause is permissive; you can use iden in commercial and proprietary projects with minimal restrictions, provided you include the license notice.

Quickstart

pip install iden

from iden.shard import create_json_shard
from iden.dataset import create_vanilla_dataset

shard = create_json_shard(data={"key": "value"}, uri="file:///path/to/data.json")
data = shard.get_data()

Requires Python 3.10 or later.

Verify before relying

• Performance characteristics when managing large numbers of shards or very large individual shard files.
• Memory overhead of the caching mechanism and how it scales with dataset size.
• Compatibility with distributed training frameworks beyond what the fact sheet documents.

Package facts

Evidence: iden-0.4.1-py3-none-any.whl

Keywords: dataset, shard

Tags

More Libraries packages