dvc-data

What it is and what it does

dvc-data is DVC's data management subsystem, providing the core abstractions for storing, indexing, and verifying data files. It implements hash-based file objects (HashFile) similar to Git's object model, object databases (HashFileDB) that verify contents by hash, and a trie-like index structure (DataIndex) that maps file paths to metadata and storage locations. The package abstracts storage backends through pluggable ObjectStorage and FileStorage implementations, allowing DVC to work with different underlying filesystems via fsspec.

The library is designed as a low-level building block for data pipeline tools. It handles the mechanics of content-addressed storage—storing files by their hash digest and verifying integrity—while remaining agnostic about where or how data is actually persisted. Developers typically use it indirectly through DVC itself, though it can be imported directly for custom data management workflows.

Use it for:

• Building data pipelines that need to verify file integrity using hash-based content addressing
• Implementing custom data storage backends that integrate with DVC workflows
• Creating reproducible data indexes that track file metadata and storage locations
• Developing tools that require trie-based hierarchical data organization with hash verification

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides data storage and indexing abstractions for DVC, including hash-based file verification, object databases, and trie-like data indexes with pluggable storage backends.

Yes, if you are building on top of DVC or need hash-based file verification and content-addressed storage abstractions. The package is actively maintained, has low install friction, and carries no known vulnerabilities. However, it is marked Beta, so APIs may shift; verify compatibility with your DVC version before depending on it in production.

Install

dvc-data on PyPI

pip install dvc-data

uv add dvc-data

poetry add dvc-data

Installing dvc-data

Before you install

Low install friction with a pure-Python wheel and active maintenance; last commit 2026-08-10 with 168 days since release. Ten runtime dependencies are all established packages.

License in practice

Apache 2.0 permissive license allows use in most commercial and open-source projects without significant restrictions.

Quickstart

pip install dvc-data

from dvc_data.hashfile import HashFile, HashFileDB
from dvc_data.index import DataIndex, DataIndexEntry

index = DataIndex()
index[("foo",)] = DataIndexEntry(hash_info=hash_info, meta=meta)

Requires Python 3.9 or later; dvc-objects must be installed as a runtime dependency.

Verify before relying

• Whether HashFile and HashFileDB are stable APIs or still subject to breaking changes given Beta status
• Performance characteristics and scalability limits for large indexes or storage backends
• Compatibility guarantees with specific versions of dvc-objects and fsspec

Package facts

Evidence: dvc_data-3.18.3-py3-none-any.whl

Tags

More Database packages