---
id: iden
version: "0.4.1"
license: BSD-3-Clause
license_treatment: permissive
maintenance: active
---

# iden — simple library to manage a dataset of shards to train machine learning models

License: permissive · Maintenance: active · Downloads: 583.2K/mo

## What it is and what it does

iden is a Python library for organizing and accessing machine learning training data split into shards—discrete data chunks that can be stored in different formats and loaded on demand. It abstracts away the mechanics of managing train/validation/test splits, persisting shards to disk, and retrieving them lazily so you don't load everything into memory at once. Each shard has a URI for reproducible identification and optional caching for frequently accessed data.

The library depends on coola (for data comparison) and objectory (for dynamic object instantiation), and supports formats like JSON, YAML, Pickle, PyTorch tensors, and safetensors. It's designed for the common ML workflow where you organize data into logical splits and want to load individual shards on demand rather than materializing the entire dataset upfront.

Use it for:

- Organize large training datasets into splits (train/val/test) and load shards lazily during model training.
- Store preprocessed data in multiple formats and switch between them without rewriting shard management code.
- Cache frequently accessed shards in memory while keeping the full dataset on disk to manage memory constraints.
- Persist dataset structure and shard references using URIs for reproducible data pipelines across runs.
- Build custom shard loaders for domain-specific data formats by extending the library's extensible architecture.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

iden manages machine learning datasets organized into shards with lazy loading, supporting multiple storage formats (JSON, YAML, Pickle, PyTorch, safetensors) and optional in-memory caching.

Yes. iden is actively maintained, has no known vulnerabilities, installs with minimal friction, and solves a concrete problem in ML workflows—organizing and lazily loading sharded datasets. The permissive BSD-3-Clause license poses no restriction. The API is pre-1.0 and may change, so pin the version if stability is critical, but for new projects or exploratory work it is a solid choice.

## Install

pip install iden
uv add iden
poetry add iden

## Installing iden

Before you install: Low friction: pure Python wheel with only two runtime dependencies (coola and objectory). Actively maintained with a recent release 60 days ago and current commit activity.

License in practice: BSD-3-Clause is permissive; you can use iden in commercial and proprietary projects with minimal restrictions, provided you include the license notice.

Quickstart:

pip install iden

from iden.shard import create_json_shard
from iden.dataset import create_vanilla_dataset

shard = create_json_shard(data={"key": "value"}, uri="file:///path/to/data.json")
data = shard.get_data()

Requires Python 3.10 or later.

Verify before relying:

- Performance characteristics when managing large numbers of shards or very large individual shard files.
- Memory overhead of the caching mechanism and how it scales with dataset size.
- Compatibility with distributed training frameworks beyond what the fact sheet documents.

## Package facts

- License: BSD-3-Clause (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 583.2K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

lazy loading dataset shards, machine learning dataset management, shard-based data organization, multi-format data storage, train/val/test split management, efficient dataset caching, URI-based data persistence, dataset-management, lazy-loading, ml-training

[View on SkillFed](https://skillfed.io/packages/iden) · [View on PyPI](https://pypi.org/project/iden/)