mad-prefect

What it is and what it does

mad_prefect wraps Prefect workflow functions with a `@asset` decorator that automatically handles data persistence, caching, and querying. When you decorate a function, it becomes a data asset that can materialize its output to disk (in JSON, Parquet, or CSV), reuse cached results within a configurable expiration window, and query the stored data using DuckDB without loading it entirely into memory.

The library is built on seven runtime dependencies: prefect (the orchestration framework), duckdb (for SQL querying), pandas and pyarrow (for data handling), fsspec and sshfs (for filesystem abstraction), and jsonlines (for line-delimited JSON support). It targets Python 3.11 and 3.12, and is actively maintained. Use it to build modular ETL pipelines where each step is independently executable and queryable, or to cache intermediate results in machine learning workflows.

Use it for:

• Build ETL pipelines where each transformation step is a queryable data asset that can be executed and cached independently.
• Cache intermediate machine learning datasets and query them for model training without recomputing or loading full data into memory.
• Enable data analysts to query preprocessed assets via SQL without understanding the underlying retrieval logic.
• Maintain historical snapshots of data assets for auditing and rollback in incremental data processing workflows.
• Reduce redundant computations in multi-step workflows by reusing cached artifacts within configurable expiration windows.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Adds a data asset pattern to Prefect workflows, enabling functions to persist, cache, and query data using DuckDB with minimal boilerplate.

Yes, with conditions. The package solves a real problem—adding lightweight data asset management to Prefect—and has low install friction with active maintenance. However, the unclear license status is a blocker for any project with strict licensing requirements; resolve that first. Once licensed, it's a solid choice for Prefect users building data pipelines that benefit from caching, querying, and artifact management.

Install

mad-prefect on PyPI

pip install mad-prefect

uv add mad-prefect

poetry add mad-prefect

Installing mad-prefect

Before you install

Low install friction; wheel-only distribution. Depends on seven runtime packages including prefect, duckdb, and fsspec. Actively maintained with a release within the last 126 days.

License in practice

License status is unclear—no SPDX identifier or raw license text provided. Verify the actual license terms before adopting in proprietary or copyleft-sensitive projects.

Quickstart

pip install mad-prefect

from mad_prefect.data_assets import asset
from datetime import timedelta

@asset(path="/data/results.json", cache_expiration=timedelta(hours=1))
def generate_data():
    return [{"id": 1, "value": "a"}]

result = await generate_data()
query_result = await generate_data.query("WHERE id > 0")

Requires Python 3.11 or 3.12 (capped below 3.13); async/await syntax requires an async context.

Verify before relying

• Whether the unclear license is a documentation gap or a genuine licensing issue requiring clarification before use.
• Performance characteristics and memory footprint when querying large datasets through DuckDB.
• Compatibility with Prefect versions beyond what runtime dependency pinning may specify.

Package facts

Evidence: mad_prefect-2.3.0-py3-none-any.whl

Tags

More Application Frameworks packages