deltalite

What it is and what it does

deltalite is a Python library that performs insert-or-replace operations on Delta Lake tables using a streaming merge engine designed to keep memory usage bounded by the incoming batch size rather than the target table size. It wraps delta-rs for storage and protocol handling (transaction log, checkpoints, Parquet writing, S3 commits) but replaces the merge execution layer with a row-group streaming approach: it builds a hash set from the source batch, streams the target table one Parquet row group at a time, drops rows whose primary key matches the source, and writes survivors plus source rows into new files in a single atomic Delta commit.

The package is built as a compiled extension (cp312-abi3 wheels for Python 3.12+) and requires no Rust toolchain to install. It exposes a simple API: `DeltaLiteTable.open()` to connect to a table, `.upsert()` to perform the merge, and methods to inspect table state, schema, and history. It includes operational knobs (concurrency limits, buffer sizes, pruning strategies) both per-call and process-global via environment variables, and emits metrics via the Rust metrics facade. The primary use case is incremental sync workloads where a small batch is merged into a large, slowly-changing table, avoiding the memory deadlock and OOM risks that delta-rs MERGE can encounter.

Use it for

• Incremental event ingestion: merge small daily batches into a large partitioned events table without memory scaling to table size.
• CDC pipeline: apply change-data-capture batches to a slowly-changing dimension table with bounded memory and atomic partition commits.
• Time-series upsert: replace or insert rows by timestamp and entity ID in a large time-series table partitioned by day or hour.
• Data warehouse refresh: upsert a small batch of updated records into a large fact table with exact file-level pruning to minimize I/O.
• Multi-tenant sync: merge tenant-specific batches into a shared partitioned table where each tenant's partition is touched atomically.

Worth the install?

AI-flagged interpretation of the facts on this page. Verify before relying on it.

Install

deltalite on PyPI

Before you install

Medium install friction due to compiled wheels (cp312-abi3 for Python 3.12+), but prebuilt binaries are available for common platforms (manylinux, musllinux, macOS) on x86_64 and aarch64, so no Rust toolchain is required. Package is actively maintained with recent releases.

Requires Python 3.12 or newer; target Delta table must not use deletion vectors or column mapping features.

License in practice

MIT license permits commercial and private use with minimal restrictions; you may use, modify, and distribute deltalite freely provided you include the license notice.

Quickstart

import deltalite

table = deltalite.DeltaLiteTable.open("s3://bucket/my_table")
stats = table.upsert(record_batch, primary_keys=["id"], partition_key="day")
print(f"v{stats.version}: +{stats.rows_inserted} / ~{stats.rows_updated}")

Package facts

Evidence: deltalite-0.1.6-cp312-abi3-macosx_10_12_x86_64.whl; deltalite-0.1.6-cp312-abi3-macosx_11_0_arm64.whl; deltalite-0.1.6-cp312-abi3-manylinux_2_28_aarch64.whl; deltalite-0.1.6-cp312-abi3-manylinux_2_28_x86_64.whl; deltalite-0.1.6-cp312-abi3-musllinux_1_2_aarch64.whl; deltalite-0.1.6-cp312-abi3-musllinux_1_2_x86_64.whl

Tags

More Database packages

See also delta-spark · hops-deltalake · deltalake · delta-sharing · dbl-discoverx · delta-kernel-rust-sharing-wrapper · databricks-zerobus-ingest-sdk · mltable · pangres · apache-flink-libraries