{"categories":[{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing/2"}],"enrichment":{"capability":"RepartiPy calculates optimal partition counts for PySpark DataFrames and estimates their size using execution plan statistics, supporting both memory-cached and disk-based sampling approaches.","skillfed_tags":["pyspark","data-engineering","partition-optimization"],"use_cases":["Dynamically repartition a large DataFrame to a target partition size without manually calculating partition counts.","Estimate DataFrame size more accurately when planning cluster resource allocation or job optimization.","Repartition memory-constrained clusters using SamplingSizeEstimator to avoid caching the entire DataFrame in memory.","Reduce re-reads during repartitioning by using disk-based sampling to infer partition statistics.","Tune partition behavior by leveraging execution plan statistics during the sampling phase."],"what_it_does":"RepartiPy solves the problem of determining how many partitions a DataFrame should have without pre-computing its total size. It provides two strategies: SizeEstimator, which caches the entire DataFrame in memory and extracts partition statistics from Spark's execution plan, and SamplingSizeEstimator, which uses disk I/O (HDFS write-and-reread) to estimate size when memory is constrained. Both methods aim to be more accurate than Spark's native SizeEstimator by leveraging execution plan statistics.\n\nThe package depends on typing-extensions and packaging, has low install friction, and supports Python 3.7 through 3.12. The tradeoff is a small performance overhead\u2014benchmarks show roughly 1\u20132 minutes added to typical repartition jobs\u2014and for SamplingSizeEstimator, a requirement for HDFS and disk space. The package has been dormant since its single release on 2024-03-08, so no active maintenance or updates are expected.","worth_installing":"Yes, if you regularly repartition large DataFrames and need dynamic partition sizing without pre-computing total size. The low install friction, permissive Apache-2.0 license, and two estimation strategies make it practical. However, the dormant maintenance status since 2024-03-08 means no bug fixes or compatibility updates are forthcoming\u2014install only if you can tolerate a static dependency and are comfortable troubleshooting issues independently."},"id":"repartipy","links":{"html":"https://skillfed.io/packages/repartipy","md":"https://skillfed.io/packages/repartipy.md","pypi":"https://pypi.org/project/repartipy/"},"maintenance":{"status":"dormant"},"meta":{"latest_release":"2024-03-08","license_spdx":null,"license_treatment":"permissive","name":"repartipy","python_support":"supports_current","summary":"Helper for handling PySpark DataFrame partition size \ud83d\udcd1\ud83c\udf9b\ufe0f"},"popularity":{"monthly_downloads":171315,"position":10372,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.1.8"}