repartipy
Helper for handling PySpark DataFrame partition size 📑🎛️
Decision gist · record as of 2026-08-14
Yes, if you regularly repartition large DataFrames and need dynamic partition sizing without pre-computing total size. The low install friction, permissive Apache-2.0 license, and two estimation strategies make it practical. However, the dormant maintenance status since 2024-03-08 means no bug fixes or compatibility updates are forthcoming—install only if you can tolerate a static dependency and are comfortable troubleshooting issues independently.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires an active Spark session and, for SamplingSizeEstimator, HDFS configuration and sufficient disk space on the cluster.
- Low install friction with only two lightweight runtime dependencies (typing-extensions, packaging).
- The package is dormant since its single release on 2024-03-08, with no recent commits or maintenance activity, so expect no ongoing updates or bug fixes.
License · maintenance · safety
Apache-2.0 (permissive) — Licensed under Apache-2.0 (permissive), allowing commercial use, modification, and distribution with minimal restrictions.
last release 2024-03-08 (889 days) · last repo commit 2024-03-08 · 12 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 171,315 downloads/mo, #10,372 on PyPI
Alternatives
Verify before relying
pip install repartipy
import repartipy
with repartipy.SizeEstimator(spark=spark, df=df) as se:
partition_count = se.get_desired_partition_count(desired_partition_size_in_bytes=1073741824)
se.reproduce().repartition(partition_count).write.save("/output/path")- Accuracy improvement over Spark's native SizeEstimator in real-world scenarios beyond the documented benchmark cases.
- Whether the dormant status and lack of maintenance since March 2024 will affect compatibility with newer Spark versions.
- Performance overhead specifics for DataFrames smaller or larger than the benchmarked sizes (256 MiB and 241 GiB).
What it is and what it does
RepartiPy solves the problem of determining how many partitions a DataFrame should have without pre-computing its total size. It provides two strategies: SizeEstimator, which caches the entire DataFrame in memory and extracts partition statistics from Spark's execution plan, and SamplingSizeEstimator, which uses disk I/O (HDFS write-and-reread) to estimate size when memory is constrained. Both methods aim to be more accurate than Spark's native SizeEstimator by leveraging execution plan statistics.
The package depends on typing-extensions and packaging, has low install friction, and supports Python 3.7 through 3.12. The tradeoff is a small performance overhead—benchmarks show roughly 1–2 minutes added to typical repartition jobs—and for SamplingSizeEstimator, a requirement for HDFS and disk space. The package has been dormant since its single release on 2024-03-08, so no active maintenance or updates are expected.
Use it for
- Dynamically repartition a large DataFrame to a target partition size without manually calculating partition counts.
- Estimate DataFrame size more accurately when planning cluster resource allocation or job optimization.
- Repartition memory-constrained clusters using SamplingSizeEstimator to avoid caching the entire DataFrame in memory.
- Reduce re-reads during repartitioning by using disk-based sampling to infer partition statistics.
- Tune partition behavior by leveraging execution plan statistics during the sampling phase.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you regularly repartition large DataFrames and need dynamic partition sizing without pre-computing total size.
The low install friction, permissive Apache-2.0 license, and two estimation strategies make it practical. However, the dormant maintenance status since 2024-03-08 means no bug fixes or compatibility updates are forthcoming—install only if you can tolerate a static dependency and are comfortable troubleshooting issues independently.
Install
repartipy on PyPI
Before you install
Low install friction with only two lightweight runtime dependencies (typing-extensions, packaging). The package is dormant since its single release on 2024-03-08, with no recent commits or maintenance activity, so expect no ongoing updates or bug fixes.
Requires an active Spark session and, for SamplingSizeEstimator, HDFS configuration and sufficient disk space on the cluster.
License in practice
Licensed under Apache-2.0 (permissive), allowing commercial use, modification, and distribution with minimal restrictions.
Quickstart
pip install repartipy
import repartipy
with repartipy.SizeEstimator(spark=spark, df=df) as se:
partition_count = se.get_desired_partition_count(desired_partition_size_in_bytes=1073741824)
se.reproduce().repartition(partition_count).write.save("/output/path")Verify before relying
- Accuracy improvement over Spark's native SizeEstimator in real-world scenarios beyond the documented benchmark cases.
- Whether the dormant status and lack of maintenance since March 2024 will affect compatibility with newer Spark versions.
- Performance overhead specifics for DataFrames smaller or larger than the benchmarked sizes (256 MiB and 241 GiB).
Package facts
| License | Apache-2.0 permissive |
| Python support | Supports the current Python release >=3.7 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 2 packagestyping-extensionspackaging |
| Maintenance | Dormant 889 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 171,315 / month, #10,372 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | License :: OSI Approved :: Apache Software LicenseProgramming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.7Programming Language :: Python :: 3.8Programming Language :: Python :: 3.9 |
Evidence: repartipy-0.1.8-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “pyspark dataframe partition optimization”
- repartipyRepartiPy calculates optimal partition counts for PySpark DataFrames…
- quinnQuinn provides helper methods for PySpark DataFrame validation,…
- pbsparkConverts protobuf messages to and from PySpark DataFrames using UDFs,…
Give your agent the search over MCP, or paste the wish link into any chat.
More Distributed Computing packages
gRPC Python is an HTTP/2-based RPC framework that enables you to define and call remote procedures across network boundaries using protocol buffers for serialization.
Install it if you need RPC communication in a distributed system or are integrating with existing gRPC services.
execnet lets you spawn and communicate with Python interpreters across local processes, remote hosts, and different platforms, using a simple API for task distribution and inter-process messaging.
However, the aging maintenance status (275 days since last release) means you should verify it meets your concurrency and performance needs before committing to a…
Cloudpickle extends Python's standard pickle module to serialize lambda functions, interactively-defined functions and classes, and other constructs that the default pickle cannot handle, making it suitable for cluster computing and remote code execution.
Install it if you need to serialize lambda functions, interactively-defined code, or non-standard Python constructs for cluster computing or distributed execution.
Provides a unified, open()-compatible Python API for streaming large files from remote storage (S3, GCS, Azure, HDFS, SFTP, HTTP) and local filesystems, with transparent compression support.
Install it if you work with large files on cloud storage or remote systems and want to avoid writing boilerplate around multiple SDKs.
Portalocker provides cross-platform file locking with support for exclusive and shared locks, plus Redis-based distributed locks and process-aware PID file locking.
Install it if you need file or process coordination; the optional extras (pywin32, redis) are only required for specific lock types.
Ray is a distributed computing framework that scales Python applications from a single machine to multi-node clusters, providing abstractions for parallel tasks, stateful actors, and shared objects.
See also pyspark-pandas · pyspark-client · pyspark · spark-sklearn · spark-expectations · pyspark-test · pyspark-huggingface · koalas · graphframes-py · mrmr-selection