loky

What it is and what it does

Loky is a robust implementation of Python's ProcessPoolExecutor that addresses known limitations in the standard library's multiprocessing and concurrent.futures modules. It uses fork + exec on POSIX systems instead of fork alone, which prevents crashes when third-party libraries (like OpenMP or macOS Accelerate) are involved. The package integrates cloudpickle transparently, allowing you to submit interactively defined functions and lambdas to worker processes without the typical pickling constraints.

The core feature is the reusable executor pattern: instead of spawning and tearing down a new executor for each batch of tasks, you can maintain a singleton instance that persists across calls, with optional automatic worker shutdown after idle timeouts. This avoids repeated spawning overhead. Loky also eliminates the need for `if __name__ == "__main__":` guards on Windows by leveraging cloudpickle to serialize functions from the `__main__` module, and it implements deadlock-free handling of worker process crashes with meaningful error reporting.

Use it for:

• Running CPU-bound tasks in parallel across multiple cores without fork-only multiprocessing hazards.
• Submitting lambda functions and locally defined functions to worker processes in interactive environments.
• Maintaining a persistent pool of workers across multiple independent parallel workloads to reduce spawn overhead.
• Executing tasks on macOS or with OpenMP-linked libraries where standard multiprocessing.Pool causes crashes.
• Batch processing workflows that need automatic worker cleanup after periods of inactivity.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Loky provides a reusable ProcessPoolExecutor that handles parallel task execution across multiple worker processes with safer spawning behavior and transparent support for non-picklable functions.

Yes. Loky is actively maintained, has no known vulnerabilities, and solves real problems in the standard library's process pool implementation—particularly safer spawning behavior and reusability. The single dependency (cloudpickle) is lightweight. Install it if you need robust parallel task execution, especially on macOS or with third-party C libraries, or if you want to avoid repeated executor spawn overhead.

Install

loky on PyPI

pip install loky

uv add loky

poetry add loky

Installing loky

Before you install

Low friction: pure Python wheel with a single runtime dependency (cloudpickle). Actively maintained with recent commits; last release was 352 days ago and the repository shows ongoing activity.

License in practice

BSD 3-Clause License permits commercial and private use with minimal restrictions; you must retain copyright notices and disclaimers in source and binary distributions.

Quickstart

pip install loky

from loky import get_reusable_executor

def task(x):
    return x * 2

executor = get_reusable_executor(max_workers=4)
result = executor.submit(task, 5).result()
print(result)

Requires Python 3.9 or later.

Verify before relying

• Whether psutil is truly optional or required for production use despite being listed as optional in the description.
• Performance characteristics and overhead compared to standard concurrent.futures.ProcessPoolExecutor in Python 3.7+.

Package facts

Evidence: loky-3.5.6-py3-none-any.whl

Tags

More Libraries packages