koheesio

What it is and what it does

Koheesio is a Python framework for constructing data pipelines from reusable, step-based components. It emphasizes modularity, testability, and strong typing through Pydantic, enabling developers to build robust data tasks that can be composed into larger workflows. The framework is designed to work with multiple data processing libraries and frameworks, making it adaptable to various data scales and technologies.

Unlike workflow orchestration tools (Airflow, Luigi, Databricks), Koheesio focuses on making individual data tasks resilient, observable, and maintainable. It provides built-in logging, flexible context customization, and a foundation for data validation, transformation, and ETL work. The framework is positioned as a complement to orchestration tools rather than a replacement, allowing teams to write well-engineered data tasks that can be orchestrated separately.

Use it for:

• Build reusable ETL components that can be tested independently and composed into larger data workflows.
• Write data transformation and validation logic with strong typing and structured configuration using Pydantic.
• Create modular data tasks for use within orchestration frameworks like Apache Airflow or Databricks Workflows.
• Develop data quality checks and data processing steps with built-in logging and error handling.
• Implement step-based data pipelines where each task is a manageable, testable unit of work.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Koheesio is a Python framework for building modular, reusable data pipelines using step-based components with strong typing via Pydantic, supporting multiple data processing libraries and frameworks.

Yes. Koheesio is actively maintained, has low install friction, carries no known vulnerabilities, and is licensed permissively. It fills a clear niche for developers building modular, testable data tasks—particularly those who plan to integrate with orchestration tools. The framework's reliance on well-established dependencies (Pydantic, cryptography, PyYAML) and its production-stable classifier make it a sound choice for data engineering work.

Install

koheesio on PyPI

pip install koheesio

uv add koheesio

poetry add koheesio

Installing koheesio

Before you install

Low install friction with a pure-Python wheel. Actively maintained with a recent release (18 days ago) and active CI/CD. Seven runtime dependencies are all well-established libraries (pydantic, cryptography, pyyaml, pytz, jsonpickle, python-decouple, tomli).

License in practice

Licensed under Apache 2.0 (permissive), allowing commercial use, modification, and distribution with minimal restrictions—suitable for most organizational and open-source contexts.

Quickstart

pip install koheesio

from koheesio import Step
from pydantic import Field

class MyStep(Step):
    input_data: str = Field(..., description="Input data")
    
    def execute(self):
        self.output = self.input_data.upper()

step = MyStep(input_data="hello")
step.run()

Requires Python 3.9 or later (supports 3.9, 3.10, 3.11, 3.12; capped below 3.13).

Verify before relying

• Whether the framework integrates with specific orchestration tools (Airflow, Luigi, Databricks) beyond the stated design goal.
• Performance characteristics and scalability limits for large-scale data processing tasks.
• Availability and maturity of built-in readers, writers, and transformations mentioned in the description.

Package facts

Evidence: koheesio-0.11.0-py3-none-any.whl

Keywords: data-engineering, data-pipeline, data-processing, data-quality, data-transformation, data-validation, etl, modular, pydantic, pyspark, reusable, software-engineering, spark, step-based, workflow

Tags

More Python Modules packages