repartipy

What it is and what it does

RepartiPy solves the problem of determining how many partitions a DataFrame should have without pre-computing its total size. It provides two strategies: SizeEstimator, which caches the entire DataFrame in memory and extracts partition statistics from Spark's execution plan, and SamplingSizeEstimator, which uses disk I/O (HDFS write-and-reread) to estimate size when memory is constrained. Both methods aim to be more accurate than Spark's native SizeEstimator by leveraging execution plan statistics.

The package depends on typing-extensions and packaging, has low install friction, and supports Python 3.7 through 3.12. The tradeoff is a small performance overhead—benchmarks show roughly 1–2 minutes added to typical repartition jobs—and for SamplingSizeEstimator, a requirement for HDFS and disk space. The package has been dormant since its single release on 2024-03-08, so no active maintenance or updates are expected.

Use it for:

• Dynamically repartition a large DataFrame to a target partition size without manually calculating partition counts.
• Estimate DataFrame size more accurately when planning cluster resource allocation or job optimization.
• Repartition memory-constrained clusters using SamplingSizeEstimator to avoid caching the entire DataFrame in memory.
• Reduce re-reads during repartitioning by using disk-based sampling to infer partition statistics.
• Tune partition behavior by leveraging execution plan statistics during the sampling phase.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

RepartiPy calculates optimal partition counts for PySpark DataFrames and estimates their size using execution plan statistics, supporting both memory-cached and disk-based sampling approaches.

Yes, if you regularly repartition large DataFrames and need dynamic partition sizing without pre-computing total size. The low install friction, permissive Apache-2.0 license, and two estimation strategies make it practical. However, the dormant maintenance status since 2024-03-08 means no bug fixes or compatibility updates are forthcoming—install only if you can tolerate a static dependency and are comfortable troubleshooting issues independently.

Install

repartipy on PyPI

pip install repartipy

uv add repartipy

poetry add repartipy

Installing repartipy

Before you install

Low install friction with only two lightweight runtime dependencies (typing-extensions, packaging). The package is dormant since its single release on 2024-03-08, with no recent commits or maintenance activity, so expect no ongoing updates or bug fixes.

License in practice

Licensed under Apache-2.0 (permissive), allowing commercial use, modification, and distribution with minimal restrictions.

Quickstart

pip install repartipy

import repartipy

with repartipy.SizeEstimator(spark=spark, df=df) as se:
    partition_count = se.get_desired_partition_count(desired_partition_size_in_bytes=1073741824)
    se.reproduce().repartition(partition_count).write.save("/output/path")

Requires an active Spark session and, for SamplingSizeEstimator, HDFS configuration and sufficient disk space on the cluster.

Verify before relying

• Accuracy improvement over Spark's native SizeEstimator in real-world scenarios beyond the documented benchmark cases.
• Whether the dormant status and lack of maintenance since March 2024 will affect compatibility with newer Spark versions.
• Performance overhead specifics for DataFrames smaller or larger than the benchmarked sizes (256 MiB and 241 GiB).

Package facts

Evidence: repartipy-0.1.8-py3-none-any.whl

Keywords: apachespark, spark, pyspark

Tags

More Distributed Computing packages