sparkmeasure

What it is and what it does

SparkMeasure is a Python wrapper around a Scala-based performance instrumentation tool for Apache Spark. It lets you collect detailed runtime metrics from Spark jobs—task execution times, shuffle metrics, memory usage, and other performance indicators—either interactively in notebooks or in batch mode. The package is designed for developers and data engineers who need to diagnose slow or inefficient Spark workloads, understand resource consumption, and validate performance improvements.

The tool integrates with Spark's Listener interface to capture low-level execution events without modifying your job code significantly. It can export metrics to external systems like InfluxDB, Kafka, or Prometheus for centralized monitoring, and also serves as an educational resource for understanding how Spark's internal metrics work. Since the bulk implementation is in Scala, it runs efficiently within the JVM alongside your Spark jobs.

Use it for:

• Diagnose slow PySpark jobs by collecting task-level timing and resource metrics in a Jupyter notebook
• Batch analysis of Spark workload performance using Flight Recorder mode to capture and replay metrics
• Monitor Spark job metrics in real time and export them to InfluxDB or Prometheus for dashboarding
• Understand shuffle, I/O, and memory behavior of large-scale Spark applications
• Educational exploration of Spark's Listener interface and internal performance instrumentation

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

SparkMeasure provides a Python API to collect and analyze Apache Spark performance metrics, enabling interactive troubleshooting and batch job analysis of Spark workloads.

Yes, if you work with Apache Spark and need performance visibility. Low install friction, no runtime dependencies, active maintenance, and permissive licensing make it a practical choice. Verify that your Spark environment and use case align with its requirements before adopting.

Install

sparkmeasure on PyPI

pip install sparkmeasure

uv add sparkmeasure

poetry add sparkmeasure

Installing sparkmeasure

Before you install

Low install friction with no runtime dependencies. Active maintenance as of 2026-05-19 with 828 repository stars. Supports Python 3.9 through 3.14.

License in practice

Apache-2.0 permissive license allows commercial and private use with minimal restrictions; suitable for most projects.

Quickstart

pip install sparkmeasure

from sparkmeasure import StageMetrics
metrics = StageMetrics(spark)
metrics.begin()
# run your Spark job here
metrics.end()
metrics.print_report()

Requires Apache Spark and PySpark to be installed and a running Spark session; the Scala implementation runs within the JVM.

Verify before relying

• Whether the Scala bulk implementation requires a JVM or specific Spark installation beyond what PySpark provides
• Integration requirements or configuration steps for external monitoring systems (InfluxDB, Kafka, Prometheus)
• Whether Flight Recorder mode has specific operational or storage prerequisites

Package facts

Evidence: sparkmeasure-0.28.0-py2.py3-none-any.whl

Keywords: spark, apache spark, performance, monitoring, metrics, troubleshooting

Tags

More Python Modules packages