pypeln

What it is and what it does

Pypeln is a Python library for building concurrent data pipelines using a functional API that abstracts over processes, threads, asyncio tasks, or synchronous generators. You define stages—each with map, filter, or flat_map operations—and specify how many workers should process data at each stage, with control over queue size to manage memory. Stages are composable and can be mixed (e.g., processes feeding into threads), and they integrate with regular Python code since stages are iterables.

The library targets medium-scale data tasks where parallelism is needed but frameworks like Spark or Dask feel overkill. It exposes a familiar functional programming interface and supports a pipe operator for readable pipeline construction. However, the project is abandoned—no releases since January 2022 and last commit in July 2023—so it receives no maintenance, security updates, or compatibility fixes.

Use it for

• Process a stream of network requests or I/O-bound tasks using thread workers without managing thread pools manually.
• Build a multi-stage ETL pipeline where different stages benefit from different concurrency models (e.g., asyncio for I/O, processes for CPU-bound work).
• Debug or prototype concurrent logic using the sync module before switching to threads or processes for performance.
• Parallelize CPU-bound transformations across multiple processes while controlling memory usage per stage with maxsize limits.
• Chain functional operations (map, filter) on iterables with automatic worker distribution and queue management.

Worth the install?

AI-flagged interpretation of the facts on this page. Verify before relying on it.

Install

pypeln on PyPI

Before you install

Low install friction with only three runtime dependencies. However, the package is abandoned—last release was 2022-01-06 and last commit 2023-07-20—so expect no maintenance, bug fixes, or compatibility updates for future Python versions.

Requires Python >=3.6.2,<4.0; multiprocessing-based stages require picklable functions; asyncio stages require an event loop.

License in practice

MIT license is permissive, allowing commercial and private use with minimal restrictions, making it safe from a licensing standpoint for most projects.

Quickstart

import pypeln as pl

def add_one(x):
    return x + 1

data = range(10)
stage = pl.process.map(add_one, data, workers=3, maxsize=4)
result = list(stage)

Package facts

Evidence: pypeln-0.4.9-py3-none-any.whl

Tags

More Distributed Computing packages

See also multiprocess · multitasking · threaded · pytest-parallel · aioprocessing · mpire · pytest-run-parallel · futures · concurrent-log-handler · aiomultiprocess