metaflow-checkpoint

What it is and what it does

Metaflow Checkpoint is an experimental extension for Metaflow that adds three decorators—`@checkpoint`, `@model`, and `@huggingface_hub`—to simplify state management and model handling in long-running data processing and machine learning workflows. The `@checkpoint` decorator saves and restores step state, allowing jobs to resume from the last checkpoint after failures without manual intervention. The `@model` decorator provides a straightforward way to save and load models created within a flow, while `@huggingface_hub` enables efficient loading and caching of large models from Hugging Face Hub. All three decorators handle user isolation and version control in multi-user environments, preventing conflicts and ensuring reproducibility.

The package has no runtime dependencies and installs with low friction. It is actively maintained but carries an experimental label, meaning the API and behavior may still evolve. The license treatment is unclear, so users should verify licensing terms before adoption. The package is positioned for workflows where interruptions are costly—training jobs that run for hours or days, data pipelines that process large datasets, and scenarios where collaboration requires clear model and checkpoint versioning.

Use it for:

• Resume a multi-day machine learning training job after a hardware failure or timeout without restarting from scratch.
• Cache and version Hugging Face models across a team to avoid redundant downloads and ensure everyone uses the same model snapshot.
• Save intermediate data checkpoints in a data processing pipeline so failed steps can retry from the last known good state.
• Isolate checkpoints per user in a shared Metaflow environment to prevent accidental overwrites during concurrent runs.
• Track model lineage and versions automatically as part of a Metaflow flow without manual checkpoint file management.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides decorators for Metaflow workflows to save and resume checkpoints, manage models, and load external models from Hugging Face Hub, enabling long-running jobs to recover from failures without losing progress.

Yes, if you run long-running Metaflow workflows and need checkpoint recovery and model versioning. The low install friction and active maintenance are favorable, but verify the license before use and be aware of the experimental status—test thoroughly in non-critical workflows first. Not necessary if your jobs are short-lived or you already have external checkpoint infrastructure.

Install

metaflow-checkpoint on PyPI

pip install metaflow-checkpoint

uv add metaflow-checkpoint

poetry add metaflow-checkpoint

Installing metaflow-checkpoint

Before you install

Low install friction with no runtime dependencies. Actively maintained with a recent release; however, marked as experimental, so stability and API surface may still be evolving.

License in practice

License treatment is unclear—no SPDX identifier or raw license text is available. Verify the actual license before adopting in proprietary or restricted environments.

Quickstart

pip install metaflow-checkpoint

from metaflow import FlowSpec, step
from metaflow_checkpoint import checkpoint

class MyFlow(FlowSpec):
    @step
    @checkpoint
    def train(self):
        # Your training code here
        pass

Requires Metaflow to be installed and configured; Python version support is unspecified.

Verify before relying

• Exact Python version requirements and compatibility with specific Metaflow versions
• Whether the package has a published license or if it defaults to a specific open-source license
• API stability guarantees given the 'experimental' designation
• Performance overhead of checkpoint operations on typical workloads

Package facts

Evidence: metaflow_checkpoint-0.2.11-py2.py3-none-any.whl

Tags

More Distributed Computing packages