{"categories":[{"label":"Libraries","url":"https://skillfed.io/packages/category/software-development-libraries/5"},{"label":"Scientific/Engineering","url":"https://skillfed.io/packages/category/scientific-engineering/4"},{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/5"}],"enrichment":{"capability":"iden manages machine learning datasets organized into shards with lazy loading, supporting multiple storage formats (JSON, YAML, Pickle, PyTorch, safetensors) and optional in-memory caching.","skillfed_tags":["dataset-management","lazy-loading","ml-training"],"use_cases":["Organize large training datasets into splits (train/val/test) and load shards lazily during model training.","Store preprocessed data in multiple formats and switch between them without rewriting shard management code.","Cache frequently accessed shards in memory while keeping the full dataset on disk to manage memory constraints.","Persist dataset structure and shard references using URIs for reproducible data pipelines across runs.","Build custom shard loaders for domain-specific data formats by extending the library's extensible architecture."],"what_it_does":"iden is a Python library for organizing and accessing machine learning training data split into shards\u2014discrete data chunks that can be stored in different formats and loaded on demand. It abstracts away the mechanics of managing train/validation/test splits, persisting shards to disk, and retrieving them lazily so you don't load everything into memory at once. Each shard has a URI for reproducible identification and optional caching for frequently accessed data.\n\nThe library depends on coola (for data comparison) and objectory (for dynamic object instantiation), and supports formats like JSON, YAML, Pickle, PyTorch tensors, and safetensors. It's designed for the common ML workflow where you organize data into logical splits and want to load individual shards on demand rather than materializing the entire dataset upfront.","worth_installing":"Yes. iden is actively maintained, has no known vulnerabilities, installs with minimal friction, and solves a concrete problem in ML workflows\u2014organizing and lazily loading sharded datasets. The permissive BSD-3-Clause license poses no restriction. The API is pre-1.0 and may change, so pin the version if stability is critical, but for new projects or exploratory work it is a solid choice."},"id":"iden","links":{"html":"https://skillfed.io/packages/iden","md":"https://skillfed.io/packages/iden.md","pypi":"https://pypi.org/project/iden/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-06-15","license_spdx":"BSD-3-Clause","license_treatment":"permissive","name":"iden","python_support":"supports_current","summary":"simple library to manage a dataset of shards to train machine learning models"},"popularity":{"monthly_downloads":583159,"position":5897,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.4.1"}