---
id: dask-jobqueue
version: "0.9.0"
license: BSD 3-Clause
license_treatment: permissive
maintenance: dormant
---

# dask-jobqueue — Deploy Dask on job queuing systems like PBS, Slurm, SGE or LSF

License: permissive · Maintenance: dormant · Downloads: 128.4K/mo

## What it is and what it does

dask-jobqueue bridges Dask distributed computing and HPC job schedulers, letting you submit Dask workers as jobs to PBS, Slurm, SGE, or LSF clusters. Instead of manually launching worker processes, you define cluster parameters and the package generates and submits the appropriate job scripts to your scheduler. It depends on dask and distributed to handle the actual distributed computation.

The package is designed for researchers and engineers running compute-intensive workloads on shared HPC infrastructure. You define a cluster object, configure resource requests, and scale up or down by submitting or canceling jobs through the scheduler. It abstracts away the details of job script generation and submission, making it easier to parallelize work across a cluster without learning each scheduler's syntax.

Use it for:

- Submit a Dask cluster to a Slurm scheduler on an HPC system to parallelize large data processing tasks.
- Dynamically scale worker count on a PBS cluster by submitting additional jobs as load increases.
- Run machine learning training across multiple SGE-scheduled compute nodes using Dask.
- Integrate Dask with an existing LSF job queue to leverage existing resource allocation policies.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Deploys Dask distributed computing clusters on job queuing systems like PBS, Slurm, SGE, and LSF, automating the submission and management of worker nodes.

Yes, if you need to run Dask on an HPC cluster with a supported job scheduler and are comfortable with dormant maintenance. The package is stable and low-friction to install, but expect no active development or bug fixes. Not suitable if you require ongoing support or compatibility with very recent Dask versions.

## Install

pip install dask-jobqueue
uv add dask-jobqueue
poetry add dask-jobqueue

## Installing dask-jobqueue

Before you install: Low install friction with only two runtime dependencies (dask and distributed). Maintenance status is dormant—last release was 722 days ago—so expect no active bug fixes or feature development, though the package remains functional for its core use case.

License in practice: BSD 3-Clause permissive license allows commercial and private use with minimal restrictions; you must include the license text in distributions.

Quickstart:

pip install dask-jobqueue

from dask_jobqueue import PBSCluster
cluster = PBSCluster(n_workers=2, cores=2, memory='2GB')
cluster.scale()

Requires Python >=3.10 and access to a job queuing system (PBS, Slurm, SGE, or LSF) on the target HPC environment.

Verify before relying:

- Whether dormant status means critical bugs remain unfixed or if the package is stable enough for production HPC use.
- Compatibility with recent versions of dask and distributed beyond what the fact sheet indicates.

## Package facts

- License: BSD 3-Clause (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: dormant
- Downloads: 128.4K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

dask job queue deployment, slurm dask cluster, pbs distributed computing, hpc job scheduler dask, dask worker submission, cluster management dask, hpc-cluster, job-scheduler

[View on SkillFed](https://skillfed.io/packages/dask-jobqueue) · [View on PyPI](https://pypi.org/project/dask-jobqueue/)