bounded-pool-executor

What it is and what it does

bounded-pool-executor provides drop-in replacements for concurrent.futures.ProcessPoolExecutor and ThreadPoolExecutor that apply backpressure to task submission. Instead of accepting all submitted tasks into an unbounded queue, it uses a BoundedSemaphore to block submit() calls until a worker finishes, ensuring the queue never grows beyond the number of active workers. This prevents the memory exhaustion that occurs when processing millions of items with standard executors on memory-constrained systems.

The package is a thin wrapper around the standard library's executor classes, requiring no external dependencies. It is designed as a direct replacement: you import BoundedProcessPoolExecutor or BoundedThreadPoolExecutor and use them identically to their standard counterparts, but with the guarantee that memory usage stays bounded by the number of workers rather than the number of submitted tasks.

Use it for:

• Processing millions of items on a memory-constrained system without exhausting available RAM.
• Batch processing large datasets where task submission rate far exceeds worker completion rate.
• Long-running services that accept unbounded work queues and need to apply backpressure to prevent memory issues.
• Distributed crawling or scraping tasks where the task generator runs faster than workers can consume.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Wraps Python's ProcessPoolExecutor and ThreadPoolExecutor to limit queued tasks, preventing memory exhaustion when submitting millions of jobs by only enqueuing new tasks as workers become available.

Yes, if you are processing large task volumes on memory-limited systems and can tolerate dormant maintenance. The package solves a real problem (memory exhaustion with standard executors) and has no dependencies, making it low-risk to add. However, verify compatibility with your Python version first, as the latest release was 2019-06-04 and no updates are planned.

Install

bounded-pool-executor on PyPI

pip install bounded-pool-executor

uv add bounded-pool-executor

poetry add bounded-pool-executor

Installing bounded-pool-executor

Before you install

Low install friction with no runtime dependencies. Maintenance is dormant—last release was 2019-06-04 and last commit 2024-03-08—so expect no active support or updates, though the repository remains active.

License in practice

MIT license is permissive, allowing commercial and private use with minimal restrictions; attribution required but no copyleft obligations.

Quickstart

from bounded_pool_executor import BoundedProcessPoolExecutor

def task(item):
    return item

with BoundedProcessPoolExecutor(max_workers=5) as executor:
    for i in range(10000):
        executor.submit(task, i)

No external dependencies, but verify Python version compatibility given latest release was 2019-06-04.

Verify before relying

• Whether the package works with current Python versions given dormant maintenance status since latest release.
• Whether BoundedThreadPoolExecutor is also provided or only BoundedProcessPoolExecutor.
• Performance characteristics and overhead of the bounded semaphore approach vs. standard executors.

Package facts

Evidence: bounded_pool_executor-0.0.3-py3-none-any.whl

Keywords: concurrent, futures, ProcessPoolExecutor, ThreadPoolExecutor, Semaphore, memory, leak

Tags

More Distributed Computing packages