{"categories":[{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing/3"}],"enrichment":{"capability":"Deploys Dask distributed computing clusters on job queuing systems like PBS, Slurm, SGE, and LSF, automating the submission and management of worker nodes.","skillfed_tags":["hpc-cluster","job-scheduler"],"use_cases":["Submit a Dask cluster to a Slurm scheduler on an HPC system to parallelize large data processing tasks.","Dynamically scale worker count on a PBS cluster by submitting additional jobs as load increases.","Run machine learning training across multiple SGE-scheduled compute nodes using Dask.","Integrate Dask with an existing LSF job queue to leverage existing resource allocation policies."],"what_it_does":"dask-jobqueue bridges Dask distributed computing and HPC job schedulers, letting you submit Dask workers as jobs to PBS, Slurm, SGE, or LSF clusters. Instead of manually launching worker processes, you define cluster parameters and the package generates and submits the appropriate job scripts to your scheduler. It depends on dask and distributed to handle the actual distributed computation.\n\nThe package is designed for researchers and engineers running compute-intensive workloads on shared HPC infrastructure. You define a cluster object, configure resource requests, and scale up or down by submitting or canceling jobs through the scheduler. It abstracts away the details of job script generation and submission, making it easier to parallelize work across a cluster without learning each scheduler's syntax.","worth_installing":"Yes, if you need to run Dask on an HPC cluster with a supported job scheduler and are comfortable with dormant maintenance. The package is stable and low-friction to install, but expect no active development or bug fixes. Not suitable if you require ongoing support or compatibility with very recent Dask versions."},"id":"dask-jobqueue","links":{"html":"https://skillfed.io/packages/dask-jobqueue","md":"https://skillfed.io/packages/dask-jobqueue.md","pypi":"https://pypi.org/project/dask-jobqueue/"},"maintenance":{"status":"dormant"},"meta":{"latest_release":"2024-08-22","license_spdx":null,"license_treatment":"permissive","name":"dask-jobqueue","python_support":"supports_current","summary":"Deploy Dask on job queuing systems like PBS, Slurm, SGE or LSF"},"popularity":{"monthly_downloads":128449,"position":11706,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.9.0"}