{"categories":[{"label":"Database","url":"https://skillfed.io/packages/category/database/4"}],"enrichment":{"capability":"Performs streaming, partition-level upsert operations on Delta Lake tables with bounded memory usage, replacing delta-rs's SQL MERGE with a merge engine that scales with batch size rather than target table size.","skillfed_tags":["delta-lake","streaming-merge","incremental-sync"],"use_cases":["Incremental event ingestion: merge small daily batches into a large partitioned events table without memory scaling to table size.","CDC pipeline: apply change-data-capture batches to a slowly-changing dimension table with bounded memory and atomic partition commits.","Time-series upsert: replace or insert rows by timestamp and entity ID in a large time-series table partitioned by day or hour.","Data warehouse refresh: upsert a small batch of updated records into a large fact table with exact file-level pruning to minimize I/O.","Multi-tenant sync: merge tenant-specific batches into a shared partitioned table where each tenant's partition is touched atomically."],"what_it_does":"deltalite is a Python library that performs insert-or-replace operations on Delta Lake tables using a streaming merge engine designed to keep memory usage bounded by the incoming batch size rather than the target table size. It wraps delta-rs for storage and protocol handling (transaction log, checkpoints, Parquet writing, S3 commits) but replaces the merge execution layer with a row-group streaming approach: it builds a hash set from the source batch, streams the target table one Parquet row group at a time, drops rows whose primary key matches the source, and writes survivors plus source rows into new files in a single atomic Delta commit.\n\nThe package is built as a compiled extension (cp312-abi3 wheels for Python 3.12+) and requires no Rust toolchain to install. It exposes a simple API: `DeltaLiteTable.open()` to connect to a table, `.upsert()` to perform the merge, and methods to inspect table state, schema, and history. It includes operational knobs (concurrency limits, buffer sizes, pruning strategies) both per-call and process-global via environment variables, and emits metrics via the Rust metrics facade. The primary use case is incremental sync workloads where a small batch is merged into a large, slowly-changing table, avoiding the memory deadlock and OOM risks that delta-rs MERGE can encounter.","worth_installing":"Yes, if you are merging small batches into large Delta Lake tables and need predictable, bounded memory usage. The package is actively maintained, has no known vulnerabilities, uses permissive MIT licensing, and solves a real problem (delta-rs MERGE memory scaling and deadlock risk). Install friction is medium but manageable with prebuilt wheels. Not suitable if your table uses deletion vectors or column mapping, or if you need SCD2 semantics."},"id":"deltalite","links":{"html":"https://skillfed.io/packages/deltalite","md":"https://skillfed.io/packages/deltalite.md","pypi":"https://pypi.org/project/deltalite/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-10","license_spdx":"MIT","license_treatment":"permissive","name":"deltalite","python_support":"supports_current","summary":"Streaming partition upsert for Delta tables, replacing delta-rs SQL MERGE"},"popularity":{"monthly_downloads":176329,"position":10248,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.1.6"}