databricks-zerobus-ingest-sdk

What it is and what it does

The Databricks Zerobus Ingest SDK is a thin Python wrapper around a native Rust implementation (via PyO3 bindings) for streaming records into Databricks Delta tables. It provides both synchronous and asynchronous APIs, handles OAuth 2.0 authentication automatically, and supports two serialization formats: JSON (simpler, higher per-record overhead) and Protocol Buffers (compact, type-safe, recommended for production). The SDK manages stream lifecycle, automatic recovery on transient failures, and acknowledgment tracking—you ingest records in a loop and call flush() to confirm durability, or register callbacks to be notified as records commit.

Core ingestion (gRPC, OAuth, stream management) runs in the compiled Rust layer, delivering native performance. The package requires Python 3.9–3.14 and depends on protobuf and requests at runtime. Optional Arrow Flight ingestion is available via the [arrow] extra, which installs pyarrow with version selection for your Python version. The SDK is marked Production/Stable and released actively.

Use it for:

• Stream sensor or IoT telemetry data into Delta tables for real-time analytics.
• Ingest application logs or events from multiple sources into a centralized Delta table.
• Build a data pipeline that continuously appends records to Delta with automatic acknowledgment and recovery.
• Prototype quick data ingestion workflows using JSON before migrating to Protocol Buffers for production throughput.
• Integrate Zerobus ingestion into async Python applications without blocking.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

A Python client for streaming data ingestion into Databricks Delta tables via the Zerobus service, supporting both JSON and Protocol Buffer serialization with synchronous and asynchronous APIs.

Yes, if you are ingesting data into Databricks Delta tables and want a native-performance Python client with built-in recovery and acknowledgment tracking. The medium install friction (compiled wheels) is typical for performance-critical packages. License treatment is unclear—verify the actual license before use in proprietary or restricted contexts. No known vulnerabilities. Active maintenance and broad Python version support (3.9–3.14) make it production-ready.

Install

databricks-zerobus-ingest-sdk on PyPI

pip install databricks-zerobus-ingest-sdk

uv add databricks-zerobus-ingest-sdk

poetry add databricks-zerobus-ingest-sdk

Installing databricks-zerobus-ingest-sdk

Before you install

Medium install friction due to compiled wheels (abi3 bindings); active maintenance with release 1 day old. Supports Python 3.9–3.14 on Linux (x86_64, aarch64), macOS, and Windows. Two lightweight runtime dependencies (protobuf, requests).

Quickstart

pip install databricks-zerobus-ingest-sdk

from zerobus.sdk.sync import ZerobusSdk
from zerobus.sdk.shared import RecordType, StreamConfigurationOptions, TableProperties

sdk = ZerobusSdk(server_endpoint, workspace_url)
table_properties = TableProperties("main.default.air_quality")
options = StreamConfigurationOptions(record_type=RecordType.JSON)
stream = sdk.create_stream(client_id, client_secret, table_properties, options)
stream.ingest_record_offset({"device_name": "sensor-1", "temp": 20})
stream.flush()
stream.close()

Requires Python 3.9–3.14; free-threaded builds (3.14t) are not supported. Workspace setup, table creation, and service principal credentials required.

Verify before relying

• Whether the package's license is actually unspecified or simply not declared in PyPI metadata.
• Whether free-threaded Python builds (e.g., 3.14t) will be supported in future releases.
• Performance characteristics and throughput benchmarks compared to alternative ingestion methods.

Package facts

Evidence: databricks_zerobus_ingest_sdk-1.6.1-cp39-abi3-manylinux_2_34_aarch64.whl; databricks_zerobus_ingest_sdk-1.6.1-cp39-abi3-manylinux_2_34_x86_64.whl; databricks_zerobus_ingest_sdk-1.6.1-cp39-abi3-win_amd64.whl

Keywords: zerobus, databricks, sdk

Tags

More Database packages