mapply

What it is and what it does

mapply wraps Pandas' apply method to run operations in parallel across multiple CPU cores. Instead of processing a DataFrame sequentially, it splits the work into chunks, assigns them to worker processes from a shared queue, and collects results—allowing irregular workloads to finish faster when some chunks are more expensive than others. It uses pathos for multiprocessing (which handles complex Python objects better than the standard library) and tqdm for progress tracking.

The package sits between lightweight (but rigid) solutions like pandarallel and heavier frameworks like Dask. You configure it once with init(), specifying worker count, chunk size, and queue depth, then call .mapply() on Series or DataFrame columns as you would .apply(). It avoids unnecessary multiprocessing overhead for small datasets and lets you tune parallelism for your specific workload.

Use it for:

• Apply expensive transformations (e.g., NLP, image processing) to millions of rows in a DataFrame without rewriting code.
• Speed up feature engineering pipelines where different rows have variable computation cost.
• Process large CSV or database query results in parallel without converting to Dask or Spark.
• Parallelize custom Python functions on Pandas Series when Numba is not applicable.
• Benchmark and tune multiprocessing behavior for your specific hardware and workload.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides a lightweight, customizable multi-core apply function for Pandas DataFrames that distributes work across CPU cores using a shared queue for load balancing.

Yes, if you apply custom Python functions to large Pandas DataFrames and want a lightweight, tunable alternative to Dask. The low dependency count, active maintenance, and permissive license make it a safe choice. Requires Python 3.11+; verify it works on your OS (fork vs. spawn behavior) and that your workload benefits from parallelization (overhead is not free for small datasets).

Install

mapply on PyPI

pip install mapply

uv add mapply

poetry add mapply

Installing mapply

Before you install

Low friction: pure Python wheel with five runtime dependencies (pathos, multiprocess, psutil, tqdm, pandas). Actively maintained with recent commits and stable production status.

License in practice

BSD-3-Clause is permissive; you may use, modify, and distribute this package freely in commercial and private projects with minimal restrictions.

Quickstart

pip install mapply

import pandas as pd
import mapply

mapply.init(n_workers=-1, chunk_size=100)
df = pd.DataFrame({"A": list(range(100))})
df["squared"] = df.A.mapply(lambda x: x**2)

Requires Python 3.11 or later.

Verify before relying

• Whether the package works reliably on Windows (fork vs. spawn multiprocessing behavior on different OS).
• Performance gains relative to Pandas' built-in Numba engine for specific workload types.
• Memory overhead when processing very large DataFrames with many workers.

Package facts

Evidence: mapply-0.2.0-py3-none-any.whl

Tags

More Python Modules packages