luigi
Workflow mgmgt + task scheduling + dependency resolution.
Decision gist · record as of 2026-08-14
Yes. Luigi is a mature, actively maintained framework with no known vulnerabilities, permissive licensing, and low installation friction. It's well-suited if you're building multi-stage batch pipelines or orchestrating interdependent long-running jobs. The web UI and dependency visualization are valuable for monitoring complex workflows. Install it if you need workflow orchestration; skip it if your tasks are simple, one-off scripts.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Installation is straightforward with low friction.
- The package is actively maintained with a recent release and has been in production use since 2012, with an active repository showing ongoing development.
License · maintenance · safety
permissive license (permissive) — Licensed under Apache License 2.0 (permissive), which allows commercial and private use with minimal restrictions—you may use, modify, and distribute the software freely provided you include license notices.
last release 2026-05-07 (99 days) · last repo commit 2026-07-18 · 18,763 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 1,273,178 downloads/mo, #4,128 on PyPI
Alternatives
Verify before relying
pip install luigi
import luigi
class MyTask(luigi.Task):
def output(self):
return luigi.LocalTarget('output.txt')
def run(self):
with self.output().open('w') as f:
f.write('done')
if __name__ == '__main__':
luigi.build([MyTask()], local_scheduler=True)- Whether the web-based visualizer requires additional dependencies or configuration beyond the base install
- Performance characteristics and scalability limits for very large dependency graphs (thousands of tasks)
What it is and what it does
Luigi is a Python workflow framework that lets you define complex pipelines as directed acyclic graphs (DAGs) of interdependent tasks. Instead of writing shell scripts or cron jobs to manage batch processes, you define tasks in Python code and specify their dependencies; Luigi automatically resolves the execution order, handles retries on failure, and ensures atomic file operations so partial results don't corrupt your data.
The framework comes with built-in support for common patterns like Hadoop jobs, Hive queries, Spark jobs, and database operations, plus a web-based dashboard for monitoring and visualizing running pipelines. It's designed for long-running batch work—the kind that takes hours or days to complete—and has been used in production at scale by organizations like Spotify, Stripe, and Foursquare to manage thousands of daily tasks.
Use it for
- Orchestrate multi-stage data pipelines combining Hadoop jobs, database dumps, and Python scripts with automatic retry and failure handling
- Build ETL workflows that extract data from multiple sources, transform it through dependent stages, and load results into data warehouses
- Manage machine learning training pipelines where feature engineering, model training, and evaluation stages must run in strict dependency order
- Schedule and monitor recurring batch jobs (reports, analytics, data aggregations) with visibility into which tasks succeeded, failed, or are pending
- Chain together heterogeneous tasks—some local Python code, some remote Spark jobs, some database operations—into a single unified workflow
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
Luigi is a mature, actively maintained framework with no known vulnerabilities, permissive licensing, and low installation friction. It's well-suited if you're building multi-stage batch pipelines or orchestrating interdependent long-running jobs. The web UI and dependency visualization are valuable for monitoring complex workflows. Install it if you need workflow orchestration; skip it if your tasks are simple, one-off scripts.
Install
luigi on PyPI
Before you install
Installation is straightforward with low friction. The package is actively maintained with a recent release and has been in production use since 2012, with an active repository showing ongoing development.
License in practice
Licensed under Apache License 2.0 (permissive), which allows commercial and private use with minimal restrictions—you may use, modify, and distribute the software freely provided you include license notices.
Quickstart
pip install luigi
import luigi
class MyTask(luigi.Task):
def output(self):
return luigi.LocalTarget('output.txt')
def run(self):
with self.output().open('w') as f:
f.write('done')
if __name__ == '__main__':
luigi.build([MyTask()], local_scheduler=True)Verify before relying
- Whether the web-based visualizer requires additional dependencies or configuration beyond the base install
- Performance characteristics and scalability limits for very large dependency graphs (thousands of tasks)
Package facts
| License | permissive license permissive |
| Python support | Supports the current Python release <3.14,>=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 5 packagespython-daemonpython-dateutiltenacitytornadotyping-extensions |
| Maintenance | Actively maintained 99 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 1,273,178 / month, #4,128 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableEnvironment :: ConsoleEnvironment :: Web EnvironmentIntended Audience :: DevelopersIntended Audience :: System AdministratorsLicense :: OSI Approved :: Apache Software LicenseProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Topic :: System :: Monitoring |
Evidence: luigi-3.8.1-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “batch job scheduling”
- luigiLuigi is a Python workflow orchestration framework that builds and…
- submititSubmitit wraps Slurm job submission and monitoring, letting you…
- azure-batchProvides a Python client library to submit, manage, and monitor…
Give your agent the search over MCP, or paste the wish link into any chat.
More Monitoring packages
Wraps any iterable to display a real-time progress bar in the terminal or Jupyter notebook, showing iteration count, elapsed time, and estimated time remaining.
Provides generated Python code for OpenTelemetry semantic conventions, enabling standardized attribute naming and constant definitions for instrumentation and telemetry collection.
Install it if you are using OpenTelemetry and want to follow semantic conventions correctly.
Provides the reference implementation of the OpenTelemetry API for collecting and exporting traces, metrics, and logs from Python applications.
Provides the abstract API and interfaces for OpenTelemetry instrumentation in Python, defining how to emit traces, metrics, and logs without tying code to a specific SDK implementation.
Exports OpenTelemetry observability data to an OpenTelemetry Collector using Protobuf-encoded messages over HTTP.
Install it if you are using OpenTelemetry in Python and need to send data to a Collector over HTTP.
Provides automatic instrumentation commands and programmatic APIs to inject distributed tracing into Python applications without code changes, detecting and instrumenting packages used by your program.
Install it if you need distributed tracing without code changes and have compatible instrumented packages in your environment.
See also b2luigi · luigi-monitor · adagio · argo-workflows · snakemake-interface-scheduler-plugins · apache-airflow-core · prefect-dbt · apache-airflow-providers-apache-pig · prefect-gitlab · awsglue-dev