dlt-meta

What it is and what it does

DLT-META bridges Databricks Lakeflow Declarative Pipelines and configuration-driven automation by reading pipeline metadata from JSON onboarding files and dynamically generating bronze and silver layer pipelines. Instead of writing pipeline code for each data source, teams define source/target metadata, data quality rules, and transformations in configuration files; a single generic pipeline then reads these specifications and orchestrates the workload. The framework supports multiple input sources (Autoloader, Delta, Eventhub, Kafka, snapshots), CDC operations, data quality expectations, quarantine tables, and custom transformations across both layers.

The package integrates with the Databricks CLI to provide interactive onboarding and deployment commands. It is designed for teams building enterprise-scale data platforms who want to templatize pipeline creation and reduce manual coding. Runtime dependencies are minimal (setuptools, databricks-sdk, PyYAML), and the package targets Python 3.8 and later.

Use it for:

• Automate bronze pipelines ingesting from multiple sources using metadata-driven configuration instead of hand-coded transformations.
• Define silver-layer transformations and data quality rules in JSON, then deploy across datasets without rewriting logic.
• Implement medallion architecture at scale by chaining bronze and silver pipelines with a single configuration parameter.
• Apply CDC and snapshot-to-stream flows declaratively by specifying them in metadata rather than coding each pattern.
• Enforce data quality expectations and quarantine invalid records automatically by declaring rules in configuration files.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

DLT-META is a metadata-driven framework that automates bronze and silver data pipelines on Databricks by reading pipeline specifications from JSON configuration files and orchestrating Lakeflow Declarative Pipeline execution.

Yes, if you are building data pipelines on Databricks and want to reduce boilerplate by adopting configuration-driven automation. The low install friction and lack of known vulnerabilities support adoption. However, the unclear license terms and aging maintenance status (332 days since last release) warrant review before committing to production use—confirm license compatibility and assess whether the slower maintenance cadence aligns with your support expectations.

Install

dlt-meta on PyPI

pip install dlt-meta

uv add dlt-meta

poetry add dlt-meta

Installing dlt-meta

Before you install

Low install friction with three straightforward runtime dependencies. Package shows aging maintenance status—last release was 332 days ago—so expect slower response to issues, though no known vulnerabilities are present.

License in practice

License treatment is unclear: the package carries a 'Databricks License' designation without a standard SPDX identifier. Review the actual license terms before adopting in commercial or redistributed contexts.

Quickstart

pip install dlt-meta

from dlt_meta import DLTMeta

# Authenticate and configure
databricks labs dlt-meta onboard
databricks labs dlt-meta deploy

Requires Databricks CLI v0.213 or later, a Databricks workspace, and Python 3.8 or higher. Must authenticate via 'databricks auth login --host WORKSPACE_HOST' before use.

Verify before relying

• Exact scope of the 'Databricks License' and whether it permits commercial use or redistribution.
• Whether the package is actively maintained or in maintenance-only mode given 332 days since last release.
• Performance characteristics and scalability limits for large pipeline configurations.

Package facts

Evidence: dlt_meta-0.0.10-py3-none-any.whl

Tags

More Testing packages