brickflows

What it is and what it does

Brickflow lets you define workflows as Python code using decorators and task dependencies, then deploy them via CLI. Instead of manually configuring jobs in a UI, you write Python functions with @wf.task decorators, specify their dependencies, and the framework translates that into workflow definitions. It handles cluster configuration, notebook execution, Spark SQL, and bash commands as tasks within a single workflow.

The package integrates with click for CLI, Jinja2 for templating, Pydantic for validation, and scheduling libraries (croniter, pendulum, cron-descriptor) to support cron-based triggers and complex task orchestration. It's aimed at teams building data pipelines who want infrastructure-as-code for their workflows, with a focus on local development and CI/CD deployment patterns.

Use it for:

• Define multi-task ETL pipelines as Python code with task dependencies, then deploy in one command.
• Parameterize workflows using common_task_parameters to inject catalog, schema, and environment variables across tasks.
• Orchestrate mixed workloads: Python tasks, notebook execution, Spark SQL, and bash commands in one definition.
• Version control workflow definitions alongside code and deploy via CI/CD pipelines.
• Schedule recurring workflows using cron expressions with automatic descriptor generation and timezone handling.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

A Python framework and CLI tool for building and deploying workflows to Databricks, enabling you to define multi-task data pipelines as Python code with task dependencies and scheduling.

Yes, if you are building data pipelines on Databricks and want to manage workflows as Python code. The package is actively maintained, has low install friction, uses permissive licensing, and integrates cleanly with standard Python tooling. It is most valuable for teams already using Python for data work and seeking a declarative, version-controllable alternative to UI-based job configuration.

Install

brickflows on PyPI

pip install brickflows

uv add brickflows

poetry add brickflows

Installing brickflows

Before you install

Low install friction with a pure-Python wheel and 14 well-established runtime dependencies. Actively maintained with a release 7 days ago and recent commits; the project has 228 stars and is in the top 5000 PyPI packages by downloads.

License in practice

Licensed under Apache-2.0 (permissive), allowing commercial use, modification, and distribution with minimal restrictions—suitable for most production environments.

Quickstart

pip install brickflows

from brickflows import Workflow, Cluster

cluster = Cluster(name="job_cluster", node_type_id="m6gd.xlarge", spark_version="13.3.x-scala2.12")
wf = Workflow("my_workflow", default_cluster=cluster)

@wf.task
def my_task():
    print("Hello")

Requires Databricks CLI installed and configured with workspace credentials (~/.databrickscfg file).

Verify before relying

• Whether the framework supports Databricks asset bundles or only legacy jobs API.
• Performance characteristics and overhead when orchestrating large numbers of tasks.
• Compatibility with Databricks Unity Catalog features beyond the common_task_parameters example.

Package facts

Evidence: brickflows-2.0.0-py3-none-any.whl

Tags

More Build Tools packages