dvc-task

What it is and what it does

dvc-task is a Celery wrapper designed for standalone Python applications that need background job execution without deploying a separate message broker or results backend. It uses Kombu's filesystem transport to store tasks and results on disk, allowing you to queue and run background processes directly from Python code without infrastructure overhead.

The package provides three main components: a ProcessManager for running and monitoring external processes as Celery tasks with persistent state (PID files, JSON metadata, stdout/stderr capture), a TemporaryWorker that consumes queued tasks and exits when idle, and a preconfigured FSApp Celery application that handles filesystem-based message brokering. It is tested on Windows despite Celery's lack of official Windows support, making it suitable for desktop and embedded Python applications.

Use it for:

• Run background processes in a desktop application without setting up Redis, RabbitMQ, or other message brokers.
• Monitor long-running external commands (shell scripts, data pipelines) with persistent state stored on the filesystem.
• Queue and execute tasks in a standalone Python script that needs to spawn worker processes on demand.
• Manage background jobs in a DVC workflow or similar data-processing pipeline without external infrastructure.
• Build a simple task queue for Windows applications where traditional Celery brokers are difficult to deploy.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

dvc-task queues and runs background jobs from Python applications using Celery, without requiring a separate messaging server or broker infrastructure.

Yes, if you need a lightweight task queue for a standalone or desktop Python application and want to avoid setting up a separate message broker. The active maintenance, low install friction, permissive license, and lack of known vulnerabilities make it a solid choice. Not suitable if you need high-volume task throughput or distributed multi-machine job orchestration; for those use cases, a traditional Celery setup with a dedicated broker is more appropriate.

Install

dvc-task on PyPI

pip install dvc-task

uv add dvc-task

poetry add dvc-task

Installing dvc-task

Before you install

Low install friction with a pure-Python wheel. Active maintenance with a recent commit on 2026-08-10. Depends on Celery 5.3+, Kombu 5.3+, funcy, shortuuid, and pywin32; the last adds Windows-specific overhead but is needed for Windows support.

License in practice

Apache-2.0 permissive license allows use in commercial and proprietary projects with minimal restrictions; you must include a copy of the license and note material changes.

Quickstart

pip install dvc-task

from dvc_task.proc import ProcessManager
from dvc_task.app import FSApp

app = FSApp(wdir=".")
manager = ProcessManager(wdir=".")
manager.run(["echo", "hello"], name="job1").delay()

Requires Celery 5.3 or later and Kombu 5.3 or later; Windows is not officially supported by Celery but dvc-task is tested on Windows.

Verify before relying

• Whether the filesystem broker scales adequately for high-volume task queues compared to traditional brokers.
• Performance characteristics of the temporary worker timeout mechanism under varying load.
• Compatibility guarantees with future Celery or Kombu releases beyond the stated minimums.

Package facts

Evidence: dvc_task-0.40.2-py3-none-any.whl

Tags

More Distributed Computing packages