openlineage-integration-common

What it is and what it does

OpenLineage-integration-common is a shared Python library that provides common building blocks for OpenLineage integrations across data platforms. Rather than each integration (Spark, Airflow, dbt, Flink, etc.) duplicating effort to model and collect metadata, this package centralizes the common data structures, utilities, and patterns needed to instrument jobs and capture lineage events. It depends on openlineage-python for core models, openlineage-sql for SQL parsing, attrs for data class support, and pyyaml for configuration handling.

The package is part of the OpenLineage ecosystem, an LF AI & Data Foundation graduate project. It enables integrations to share the effort of metadata collection rather than each project maintaining separate instrumentation logic, reducing duplication and making it easier to keep integrations in sync with new OpenLineage specification versions.

Use it for:

• Building custom OpenLineage integrations for data platforms not yet covered by official integrations.
• Standardizing metadata collection across multiple data processing frameworks in a single organization.
• Extending existing OpenLineage integrations with organization-specific facets and metadata enrichment.
• Instrumenting internal data jobs to emit lineage events to a central metadata repository like Marquez.
• Developing data observability solutions that need consistent lineage models across heterogeneous platforms.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides common Python utilities and data models for OpenLineage integrations, enabling consistent metadata and lineage collection across data platforms and jobs.

Yes, if you are building or extending OpenLineage integrations. The package is actively maintained, has no known vulnerabilities, and low install friction. Install only if you need to work with OpenLineage integration code; it is a library for integration developers rather than a general-purpose tool.

Install

openlineage-integration-common on PyPI

pip install openlineage-integration-common

uv add openlineage-integration-common

poetry add openlineage-integration-common

Installing openlineage-integration-common

Before you install

Active maintenance with a recent release (22 days old) and 2601 repository stars. Low install friction with four straightforward runtime dependencies. Supports current Python versions (3.10, 3.11, 3.12).

License in practice

Apache-2.0 permissive license allows commercial and private use with minimal restrictions, making it suitable for most integration scenarios.

Quickstart

pip install openlineage-integration-common

from openlineage.integration.common import ...
# Use common models and utilities from the integration library

Requires Python 3.10 or later.

Verify before relying

• Specific common utilities and data models exposed by this package beyond the core OpenLineage entities.
• Whether this package is intended for end-user applications or primarily for internal integration development.
• Integration-specific configuration or setup requirements beyond the standard installation.

Package facts

Evidence: openlineage_integration_common-1.52.0-py3-none-any.whl

Keywords: openlineage

Tags

More Monitoring packages