---
id: repartipy
version: "0.1.8"
license: Apache-2.0
license_treatment: permissive
maintenance: dormant
---

# repartipy — Helper for handling PySpark DataFrame partition size 📑🎛️

License: permissive · Maintenance: dormant · Downloads: 171.3K/mo

## What it is and what it does

RepartiPy solves the problem of determining how many partitions a DataFrame should have without pre-computing its total size. It provides two strategies: SizeEstimator, which caches the entire DataFrame in memory and extracts partition statistics from Spark's execution plan, and SamplingSizeEstimator, which uses disk I/O (HDFS write-and-reread) to estimate size when memory is constrained. Both methods aim to be more accurate than Spark's native SizeEstimator by leveraging execution plan statistics.

The package depends on typing-extensions and packaging, has low install friction, and supports Python 3.7 through 3.12. The tradeoff is a small performance overhead—benchmarks show roughly 1–2 minutes added to typical repartition jobs—and for SamplingSizeEstimator, a requirement for HDFS and disk space. The package has been dormant since its single release on 2024-03-08, so no active maintenance or updates are expected.

Use it for:

- Dynamically repartition a large DataFrame to a target partition size without manually calculating partition counts.
- Estimate DataFrame size more accurately when planning cluster resource allocation or job optimization.
- Repartition memory-constrained clusters using SamplingSizeEstimator to avoid caching the entire DataFrame in memory.
- Reduce re-reads during repartitioning by using disk-based sampling to infer partition statistics.
- Tune partition behavior by leveraging execution plan statistics during the sampling phase.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

RepartiPy calculates optimal partition counts for PySpark DataFrames and estimates their size using execution plan statistics, supporting both memory-cached and disk-based sampling approaches.

Yes, if you regularly repartition large DataFrames and need dynamic partition sizing without pre-computing total size. The low install friction, permissive Apache-2.0 license, and two estimation strategies make it practical. However, the dormant maintenance status since 2024-03-08 means no bug fixes or compatibility updates are forthcoming—install only if you can tolerate a static dependency and are comfortable troubleshooting issues independently.

## Install

pip install repartipy
uv add repartipy
poetry add repartipy

## Installing repartipy

Before you install: Low install friction with only two lightweight runtime dependencies (typing-extensions, packaging). The package is dormant since its single release on 2024-03-08, with no recent commits or maintenance activity, so expect no ongoing updates or bug fixes.

License in practice: Licensed under Apache-2.0 (permissive), allowing commercial use, modification, and distribution with minimal restrictions.

Quickstart:

pip install repartipy

import repartipy

with repartipy.SizeEstimator(spark=spark, df=df) as se:
    partition_count = se.get_desired_partition_count(desired_partition_size_in_bytes=1073741824)
    se.reproduce().repartition(partition_count).write.save("/output/path")

Requires an active Spark session and, for SamplingSizeEstimator, HDFS configuration and sufficient disk space on the cluster.

Verify before relying:

- Accuracy improvement over Spark's native SizeEstimator in real-world scenarios beyond the documented benchmark cases.
- Whether the dormant status and lack of maintenance since March 2024 will affect compatibility with newer Spark versions.
- Performance overhead specifics for DataFrames smaller or larger than the benchmarked sizes (256 MiB and 241 GiB).

## Package facts

- License: Apache-2.0 (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: dormant
- Downloads: 171.3K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

pyspark dataframe partition optimization, dynamic repartition without knowing size, spark dataframe size estimation, partition count calculation pyspark, spark memory-efficient repartitioning, hdfs-based dataframe sampling, pyspark, data-engineering, partition-optimization

[View on SkillFed](https://skillfed.io/packages/repartipy) · [View on PyPI](https://pypi.org/project/repartipy/)