slurm-usage

What it is and what it does

slurm-usage is a command-line tool that captures and archives SLURM job metrics before they expire from the accounting database. It queries sacct in parallel, stores raw and processed data as Parquet files, and computes efficiency metrics (CPU utilization %, memory utilization %, wasted resources) for each job. The tool is designed for long-term analysis of resource usage patterns in HPC clusters, with built-in support for incremental collection, group-based analytics, and node utilization tracking.

The package provides a CLI with commands to collect job data, analyze historical trends, display current cluster usage, and inspect node information. It uses polars for columnar data operations, pydantic for schema validation, and rich for terminal output formatting. Data is organized into raw and processed directories with a completion tracker to avoid re-fetching already-processed dates. The tool is cron-ready and supports multi-threaded parallel collection by default.

Use it for:

• Archive SLURM job metrics before the 30-day accounting purge to build a long-term efficiency baseline.
• Analyze per-user and per-group resource consumption trends to identify wasteful job patterns.
• Track node-level CPU and GPU utilization to detect hardware bottlenecks or imbalances.
• Generate automated daily reports of cluster efficiency for capacity planning and billing.
• Post-process collected Parquet data with polars for custom analytics or integration into dashboards.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Collects and preserves SLURM job efficiency metrics in Parquet format, analyzing CPU and memory usage patterns before SLURM's accounting database purges them after 30 days.

Yes, if you operate an HPC cluster with SLURM and need to preserve job metrics beyond the 30-day accounting window. The tool is actively maintained, has low install friction, and solves a real problem for long-term resource analysis. Not relevant for non-SLURM environments or clusters without accounting enabled.

Install

slurm-usage on PyPI

pip install slurm-usage

uv add slurm-usage

poetry add slurm-usage

Installing slurm-usage

Before you install

Low friction: pure Python wheel with five well-maintained dependencies (polars, pydantic, pyyaml, rich, typer). Repository is active with recent commits; last release was 326 days ago.

License in practice

MIT license is permissive; you can use, modify, and distribute this package with minimal restrictions.

Quickstart

pip install slurm-usage
slurm-usage collect --days 7
slurm-usage analyze --days 7

Requires SLURM with accounting enabled and sacct command access; Python >= 3.10.

Verify before relying

• Whether parallel collection (default 4 workers) scales linearly with cluster size or has practical limits.
• Performance characteristics when analyzing multi-year datasets in Parquet format.
• How incremental processing handles re-collection of jobs in incomplete states across large date ranges.

Package facts

Evidence: slurm_usage-3.1.0-py3-none-any.whl

Tags

More Monitoring packages