dvc-objects

What it is and what it does

DVC objects is a low-level abstraction layer that unifies access to multiple storage backends through a common interface built on fsspec. It handles the filesystem and object-database operations needed by DVC and dvc-data, supporting local storage, cloud providers (S3, Google Cloud Storage, Google Drive), and remote systems (SSH/SFTP). The package is serverless and requires only that the underlying storage support basic operations: uploading, downloading, listing, copying, and atomic renames.

This is a foundational library rather than a user-facing tool—it's designed to be used by other DVC ecosystem packages rather than directly by end users. It abstracts away the complexity of working with different storage systems, allowing higher-level tools to treat all backends uniformly.

Use it for:

• Building data pipeline tools that need to work across multiple storage backends without backend-specific code
• Implementing object-database systems that require consistent operations across local and cloud storage
• Extending DVC or dvc-data with custom storage support by leveraging the fsspec abstraction layer

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides filesystem and object-database abstractions for working with multiple storage backends (local, S3, GS, GDrive, SSH/SFTP, etc.) via fsspec, used internally by DVC and dvc-data.

Yes, if you are building tools in the DVC ecosystem or need a lightweight, actively maintained abstraction for multi-backend storage operations. The low install friction, permissive license, and active maintenance make it a safe dependency. However, this is primarily a library for other packages rather than a standalone tool—install it as a dependency, not as a direct solution to a storage problem.

Install

dvc-objects on PyPI

pip install dvc-objects

uv add dvc-objects

poetry add dvc-objects

Installing dvc-objects

Before you install

Low install friction with only two runtime dependencies (fsspec and funcy). Actively maintained with recent commits; last release was 235 days ago and the repository remains active.

License in practice

Apache 2.0 permissive license allows free use, modification, and distribution with minimal restrictions, making it suitable for both open-source and commercial projects.

Quickstart

pip install dvc-objects

from dvc_objects.fs import get_cloud_fs

fs = get_cloud_fs(None, **{"url": "s3://bucket/path"})
fs.upload("local_file", "remote_path")

Requires Python 3.9 or later; the underlying storage backend (S3, GCS, etc.) must support upload, download, list, copy, and quasiatomic rename operations.

Verify before relying

• Whether the package provides a stable public API or is primarily intended for internal DVC ecosystem use
• Performance characteristics and scalability limits for large-scale object operations
• Specific storage backend compatibility matrix and any known limitations

Package facts

Evidence: dvc_objects-5.2.0-py3-none-any.whl

Tags

More Software Development packages