acryl-datahub

What it is and what it does

Acryl-datahub is DataHub's official ingestion framework and command-line tool for metadata management. It bridges your data infrastructure and a centralized metadata catalog by supporting both pull-based connectors (scheduled crawls of databases, warehouses, BI tools, and orchestrators) and push-based APIs (programmatic metadata emission from applications and data pipelines). The package includes 50+ ready-made source connectors, built-in metadata transformers, and a Python SDK for direct integration.

You use it either as a CLI tool with YAML recipe files for scheduled batch ingestion in CI/CD pipelines, or as a Python library to emit metadata events directly from your code as data flows through your systems. It automates the capture and propagation of lineage, ownership, tags, and documentation across your data assets, feeding them into a DataHub instance for discovery and governance.

Use it for:

• Schedule automated metadata extraction from Snowflake, BigQuery, or Redshift warehouses on a daily or hourly cadence via YAML recipe.
• Emit dataset lineage and ownership metadata directly from a Python data pipeline or Spark job as it executes.
• Transform and filter metadata in transit using built-in transformers before ingestion into the catalog.
• Integrate dbt, Looker, or Airflow metadata into a centralized DataHub instance for cross-tool lineage visibility.
• Query and search the DataHub catalog programmatically or via CLI to find datasets, owners, and documentation.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

DataHub's ingestion framework and CLI for pulling metadata from 50+ data sources (Snowflake, BigQuery, dbt, Looker, Airflow, and others) into a centralized catalog, or pushing metadata programmatically from your own applications.

Yes. The package is actively maintained, has low install friction, carries no known vulnerabilities, and is licensed permissively. Install it if you need to ingest metadata from multiple data sources into DataHub or emit metadata from your own applications. It is the canonical way to connect external data systems to DataHub and is suitable for production use.

Install

acryl-datahub on PyPI

pip install acryl-datahub

uv add acryl-datahub

poetry add acryl-datahub

Installing acryl-datahub

Before you install

Low friction install with a pure-wheel distribution. Actively maintained with a release 3 days old and recent commits. Supports Python 3.10, 3.11, and 3.12.

License in practice

Licensed under Apache-2.0 (permissive), allowing commercial and private use with minimal restrictions.

Quickstart

pip install acryl-datahub

# CLI usage
datahub init --username datahub --password datahub
datahub ingest -c recipe.yml

# SDK usage
from datahub.sdk import DataHubClient, Dataset
client = DataHubClient.from_env()
dataset = Dataset(platform="snowflake", name="mydb.schema.table")
client.entities.upsert(dataset)

Requires Python 3.10 or later. DataHub server instance must be running and accessible (local or remote) for ingestion to succeed.

Verify before relying

• Whether all 50+ listed source connectors are included in the base package or require separate plugin installation.
• Performance characteristics and scalability limits for large metadata volumes or frequent push operations.
• Whether the Python SDK supports all metadata entity types and transformations available in the CLI.

Package facts

Evidence: acryl_datahub-1.7.0.3-py3-none-any.whl

Tags

More Software Development packages