hops-deltalake

What it is and what it does

hops-deltalake is a Python binding to delta-rs, the Rust implementation of Delta Lake, with built-in support for HopsFS and Pandas. It lets you read and write Delta Lake tables—the ACID-compliant data lake format built on Parquet—directly from Python without requiring Spark or Java. The package handles table versioning, file URI resolution, and conversion to Pandas DataFrames, making it suitable for data engineering workflows that need Delta Lake semantics in a lightweight Python environment.

The package is maintained as a HopsFS-compatible fork and ships prebuilt wheels for modern Python versions (3.10–3.14) on macOS arm64 and manylinux x86_64. It includes OpenTelemetry tracing support for observability. Installation is straightforward via pip on supported platforms; other architectures require a Rust compiler and maturin to build custom wheels.

Use it for:

• Read Delta Lake tables stored on S3 or HopsFS into Pandas for analysis and transformation.
• Write Pandas DataFrames as versioned Delta Lake tables with ACID guarantees and schema enforcement.
• Query Delta Lake table metadata (version history, file URIs, schema) without loading data into memory.
• Integrate Delta Lake into Python data pipelines running outside Spark or Hadoop ecosystems.
• Debug and inspect Delta Lake table structure and contents using Python for development and testing.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Reads and writes Delta Lake tables with native Python bindings backed by delta-rs, offering Pandas integration and HopsFS support for data lake operations.

Yes, if you need Delta Lake support in Python without Spark and your platform is macOS arm64 or Linux x86_64. The package is actively maintained, has no known vulnerabilities, and offers a lightweight alternative to Spark for Delta Lake operations. Install friction is moderate due to compiled wheels, but prebuilt binaries eliminate build overhead on supported platforms. Not recommended if you require other architectures or older Python versions.

Install

hops-deltalake on PyPI

pip install hops-deltalake

uv add hops-deltalake

poetry add hops-deltalake

Installing hops-deltalake

Before you install

Active maintenance with recent releases. Medium install friction due to compiled wheels (abi3 binaries for macOS arm64 and manylinux); prebuilt wheels available for Python 3.10–3.14, reducing build-from-source burden on supported platforms.

License in practice

Licensed under Apache Software License (permissive), allowing commercial and private use with minimal restrictions.

Quickstart

pip install hops-deltalake

from deltalake import DeltaTable
dt = DeltaTable("path/to/delta/table")
df = dt.to_pandas()

Requires Python ≥3.10. Prebuilt wheels available for macOS arm64 and manylinux x86_64; other platforms may require Rust compiler and maturin to build from source.

Verify before relying

• Whether HopsFS integration is required for typical use or optional for standard Delta Lake workflows.
• Performance characteristics and scalability limits compared to upstream delta-rs or Apache Spark Delta Lake.
• Stability and feature parity with the upstream delta-rs project given the HopsFS fork.

Package facts

Evidence: hops_deltalake-1.4.0.post121-cp310-abi3-macosx_11_0_arm64.whl; hops_deltalake-1.4.0.post121-cp310-abi3-manylinux_2_28_x86_64.whl

Keywords: deltalake, delta, datalake, pandas, arrow

Tags

More Database packages