---
id: sparkmeasure
version: "0.28.0"
license: Apache-2.0
license_treatment: permissive
maintenance: active
---

# sparkmeasure — Python API for sparkMeasure, a tool for performance troubleshooting of Apache Spark workloads.

License: permissive · Maintenance: active · Downloads: 2.3M/mo

## What it is and what it does

SparkMeasure is a Python wrapper around a Scala-based performance instrumentation tool for Apache Spark. It lets you collect detailed runtime metrics from Spark jobs—task execution times, shuffle metrics, memory usage, and other performance indicators—either interactively in notebooks or in batch mode. The package is designed for developers and data engineers who need to diagnose slow or inefficient Spark workloads, understand resource consumption, and validate performance improvements.

The tool integrates with Spark's Listener interface to capture low-level execution events without modifying your job code significantly. It can export metrics to external systems like InfluxDB, Kafka, or Prometheus for centralized monitoring, and also serves as an educational resource for understanding how Spark's internal metrics work. Since the bulk implementation is in Scala, it runs efficiently within the JVM alongside your Spark jobs.

Use it for:

- Diagnose slow PySpark jobs by collecting task-level timing and resource metrics in a Jupyter notebook
- Batch analysis of Spark workload performance using Flight Recorder mode to capture and replay metrics
- Monitor Spark job metrics in real time and export them to InfluxDB or Prometheus for dashboarding
- Understand shuffle, I/O, and memory behavior of large-scale Spark applications
- Educational exploration of Spark's Listener interface and internal performance instrumentation

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

SparkMeasure provides a Python API to collect and analyze Apache Spark performance metrics, enabling interactive troubleshooting and batch job analysis of Spark workloads.

Yes, if you work with Apache Spark and need performance visibility. Low install friction, no runtime dependencies, active maintenance, and permissive licensing make it a practical choice. Verify that your Spark environment and use case align with its requirements before adopting.

## Install

pip install sparkmeasure
uv add sparkmeasure
poetry add sparkmeasure

## Installing sparkmeasure

Before you install: Low install friction with no runtime dependencies. Active maintenance as of 2026-05-19 with 828 repository stars. Supports Python 3.9 through 3.14.

License in practice: Apache-2.0 permissive license allows commercial and private use with minimal restrictions; suitable for most projects.

Quickstart:

pip install sparkmeasure

from sparkmeasure import StageMetrics
metrics = StageMetrics(spark)
metrics.begin()
# run your Spark job here
metrics.end()
metrics.print_report()

Requires Apache Spark and PySpark to be installed and a running Spark session; the Scala implementation runs within the JVM.

Verify before relying:

- Whether the Scala bulk implementation requires a JVM or specific Spark installation beyond what PySpark provides
- Integration requirements or configuration steps for external monitoring systems (InfluxDB, Kafka, Prometheus)
- Whether Flight Recorder mode has specific operational or storage prerequisites

## Package facts

- License: Apache-2.0 (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 2.3M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

spark performance metrics, spark troubleshooting, spark job monitoring, spark performance analysis, spark listener python, spark metrics collection, spark workload profiling, spark-performance, metrics-collection, distributed-computing

[View on SkillFed](https://skillfed.io/packages/sparkmeasure) · [View on PyPI](https://pypi.org/project/sparkmeasure/)