pypeln

What it is and what it does

Pypeln is a Python library for building concurrent data pipelines using a functional API that abstracts over processes, threads, asyncio tasks, or synchronous generators. You define stages—each with map, filter, or flat_map operations—and specify how many workers should process data at each stage, with control over queue size to manage memory. Stages are composable and can be mixed (e.g., processes feeding into threads), and they integrate with regular Python code since stages are iterables.

The library targets medium-scale data tasks where parallelism is needed but frameworks like Spark or Dask feel overkill. It exposes a familiar functional programming interface and supports a pipe operator for readable pipeline construction. However, the project is abandoned—no releases since January 2022 and last commit in July 2023—so it receives no maintenance, security updates, or compatibility fixes.

Use it for:

• Process a stream of network requests or I/O-bound tasks using thread workers without managing thread pools manually.
• Build a multi-stage ETL pipeline where different stages benefit from different concurrency models (e.g., asyncio for I/O, processes for CPU-bound work).
• Debug or prototype concurrent logic using the sync module before switching to threads or processes for performance.
• Parallelize CPU-bound transformations across multiple processes while controlling memory usage per stage with maxsize limits.
• Chain functional operations (map, filter) on iterables with automatic worker distribution and queue management.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Pypeln lets you build concurrent data pipelines using processes, threads, or asyncio tasks with a unified functional API, allowing fine-grained control over workers and memory at each stage.

Yes, if you have a stable, medium-scale data pipeline task and can tolerate no maintenance. The library is feature-complete for its scope, has low install friction, and carries no known vulnerabilities. However, avoid it for new projects requiring long-term support or compatibility with future Python versions—the abandoned status means no bug fixes or updates. Consider it only when the problem fits its design and you accept the maintenance risk.

Install

pypeln on PyPI

pip install pypeln

uv add pypeln

poetry add pypeln

Installing pypeln

Before you install

Low install friction with only three runtime dependencies. However, the package is abandoned—last release was 2022-01-06 and last commit 2023-07-20—so expect no maintenance, bug fixes, or compatibility updates for future Python versions.

License in practice

MIT license is permissive, allowing commercial and private use with minimal restrictions, making it safe from a licensing standpoint for most projects.

Quickstart

import pypeln as pl

def add_one(x):
    return x + 1

data = range(10)
stage = pl.process.map(add_one, data, workers=3, maxsize=4)
result = list(stage)

Requires Python >=3.6.2,<4.0; multiprocessing-based stages require picklable functions; asyncio stages require an event loop.

Verify before relying

• Whether abandoned status affects real-world reliability for stable workloads or if the library is feature-complete enough to not require updates.
• Performance characteristics compared to alternatives like concurrent.futures or asyncio for specific workload types.
• Compatibility with recent Python 3.10+ versions despite no releases since 2022.

Package facts

Evidence: pypeln-0.4.9-py3-none-any.whl

Tags

More Distributed Computing packages