deltalite

What it is and what it does

deltalite is a Python library that performs insert-or-replace operations on Delta Lake tables using a streaming merge engine designed to keep memory usage bounded by the incoming batch size rather than the target table size. It wraps delta-rs for storage and protocol handling (transaction log, checkpoints, Parquet writing, S3 commits) but replaces the merge execution layer with a row-group streaming approach: it builds a hash set from the source batch, streams the target table one Parquet row group at a time, drops rows whose primary key matches the source, and writes survivors plus source rows into new files in a single atomic Delta commit.

The package is built as a compiled extension (cp312-abi3 wheels for Python 3.12+) and requires no Rust toolchain to install. It exposes a simple API: `DeltaLiteTable.open()` to connect to a table, `.upsert()` to perform the merge, and methods to inspect table state, schema, and history. It includes operational knobs (concurrency limits, buffer sizes, pruning strategies) both per-call and process-global via environment variables, and emits metrics via the Rust metrics facade. The primary use case is incremental sync workloads where a small batch is merged into a large, slowly-changing table, avoiding the memory deadlock and OOM risks that delta-rs MERGE can encounter.

Use it for:

• Incremental event ingestion: merge small daily batches into a large partitioned events table without memory scaling to table size.
• CDC pipeline: apply change-data-capture batches to a slowly-changing dimension table with bounded memory and atomic partition commits.
• Time-series upsert: replace or insert rows by timestamp and entity ID in a large time-series table partitioned by day or hour.
• Data warehouse refresh: upsert a small batch of updated records into a large fact table with exact file-level pruning to minimize I/O.
• Multi-tenant sync: merge tenant-specific batches into a shared partitioned table where each tenant's partition is touched atomically.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Performs streaming, partition-level upsert operations on Delta Lake tables with bounded memory usage, replacing delta-rs's SQL MERGE with a merge engine that scales with batch size rather than target table size.

Yes, if you are merging small batches into large Delta Lake tables and need predictable, bounded memory usage. The package is actively maintained, has no known vulnerabilities, uses permissive MIT licensing, and solves a real problem (delta-rs MERGE memory scaling and deadlock risk). Install friction is medium but manageable with prebuilt wheels. Not suitable if your table uses deletion vectors or column mapping, or if you need SCD2 semantics.

Install

deltalite on PyPI

pip install deltalite

uv add deltalite

poetry add deltalite

Installing deltalite

Before you install

Medium install friction due to compiled wheels (cp312-abi3 for Python 3.12+), but prebuilt binaries are available for common platforms (manylinux, musllinux, macOS) on x86_64 and aarch64, so no Rust toolchain is required. Package is actively maintained with recent releases.

License in practice

MIT license permits commercial and private use with minimal restrictions; you may use, modify, and distribute deltalite freely provided you include the license notice.

Quickstart

import deltalite

table = deltalite.DeltaLiteTable.open("s3://bucket/my_table")
stats = table.upsert(record_batch, primary_keys=["id"], partition_key="day")
print(f"v{stats.version}: +{stats.rows_inserted} / ~{stats.rows_updated}")

Requires Python 3.12 or newer; target Delta table must not use deletion vectors or column mapping features.

Verify before relying

• Whether the C-stream interface accepts formats beyond those explicitly named in the description.
• Behavior and performance when source batch contains duplicate primary keys (rejected per docs, but edge cases unclear).
• Compatibility with all object stores (S3, GCS, Azure, local) and whether storage_options behavior matches delta-rs conventions.

Package facts

Evidence: deltalite-0.1.6-cp312-abi3-macosx_10_12_x86_64.whl; deltalite-0.1.6-cp312-abi3-macosx_11_0_arm64.whl; deltalite-0.1.6-cp312-abi3-manylinux_2_28_aarch64.whl; deltalite-0.1.6-cp312-abi3-manylinux_2_28_x86_64.whl; deltalite-0.1.6-cp312-abi3-musllinux_1_2_aarch64.whl; deltalite-0.1.6-cp312-abi3-musllinux_1_2_x86_64.whl

Tags

More Database packages