dask-jobqueue

What it is and what it does

dask-jobqueue bridges Dask distributed computing and HPC job schedulers, letting you submit Dask workers as jobs to PBS, Slurm, SGE, or LSF clusters. Instead of manually launching worker processes, you define cluster parameters and the package generates and submits the appropriate job scripts to your scheduler. It depends on dask and distributed to handle the actual distributed computation.

The package is designed for researchers and engineers running compute-intensive workloads on shared HPC infrastructure. You define a cluster object, configure resource requests, and scale up or down by submitting or canceling jobs through the scheduler. It abstracts away the details of job script generation and submission, making it easier to parallelize work across a cluster without learning each scheduler's syntax.

Use it for:

• Submit a Dask cluster to a Slurm scheduler on an HPC system to parallelize large data processing tasks.
• Dynamically scale worker count on a PBS cluster by submitting additional jobs as load increases.
• Run machine learning training across multiple SGE-scheduled compute nodes using Dask.
• Integrate Dask with an existing LSF job queue to leverage existing resource allocation policies.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Deploys Dask distributed computing clusters on job queuing systems like PBS, Slurm, SGE, and LSF, automating the submission and management of worker nodes.

Yes, if you need to run Dask on an HPC cluster with a supported job scheduler and are comfortable with dormant maintenance. The package is stable and low-friction to install, but expect no active development or bug fixes. Not suitable if you require ongoing support or compatibility with very recent Dask versions.

Install

dask-jobqueue on PyPI

pip install dask-jobqueue

uv add dask-jobqueue

poetry add dask-jobqueue

Installing dask-jobqueue

Before you install

Low install friction with only two runtime dependencies (dask and distributed). Maintenance status is dormant—last release was 722 days ago—so expect no active bug fixes or feature development, though the package remains functional for its core use case.

License in practice

BSD 3-Clause permissive license allows commercial and private use with minimal restrictions; you must include the license text in distributions.

Quickstart

pip install dask-jobqueue

from dask_jobqueue import PBSCluster
cluster = PBSCluster(n_workers=2, cores=2, memory='2GB')
cluster.scale()

Requires Python >=3.10 and access to a job queuing system (PBS, Slurm, SGE, or LSF) on the target HPC environment.

Verify before relying

• Whether dormant status means critical bugs remain unfixed or if the package is stable enough for production HPC use.
• Compatibility with recent versions of dask and distributed beyond what the fact sheet indicates.

Package facts

Evidence: dask_jobqueue-0.9.0-py2.py3-none-any.whl

Tags

More Distributed Computing packages