dataproc-spark-connect

What it is and what it does

This package is a Python client that simplifies connecting to remote Apache Spark sessions running on Google Cloud Dataproc using the Spark Connect protocol. It wraps the Spark Connect client with additional conveniences for Dataproc-specific session management, eliminating manual setup steps and providing a fluent builder API for configuration.

You use it to create or reuse named Spark sessions in Dataproc, configure Spark properties and session templates, and execute distributed computations from Python notebooks or scripts. It handles Google Cloud authentication and session lifecycle management, and optionally integrates with Jupyter magic commands for SQL queries. The package depends on pyspark, google-cloud-dataproc, google-api-core, packaging, tqdm, and websockets.

Use it for:

• Run distributed Spark jobs on Dataproc from a local Python script or Jupyter notebook without manual session provisioning.
• Share a single Spark session across multiple notebooks by reusing named session IDs, reducing startup time and costs.
• Execute Spark SQL queries interactively in Jupyter using magic commands (with sparksql-magic installed).
• Configure Spark executor memory, cores, and Dataproc runtime versions via a fluent builder API.
• Manage session TTL and idle timeouts to control Dataproc resource lifecycle and billing.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Wraps Apache Spark Connect to let Python applications connect to remote Dataproc Spark sessions via the Spark Connect protocol without extra setup steps.

Yes. The package is actively maintained, has low install friction, carries no known vulnerabilities, and is licensed permissively. It is worth installing if you need to run Spark workloads on Google Cloud Dataproc from Python and want to avoid manual session setup. The main prerequisite is Google Cloud authentication and appropriate IAM permissions.

Install

dataproc-spark-connect on PyPI

pip install dataproc-spark-connect

uv add dataproc-spark-connect

poetry add dataproc-spark-connect

Installing dataproc-spark-connect

Before you install

Low install friction with a pure-Python wheel. Active maintenance as of 2026-04-07, with recent commits. Depends on google-cloud-dataproc, pyspark, and standard utilities; all are established packages.

License in practice

Licensed under Apache 2.0 (permissive), allowing use in commercial and private projects with minimal restrictions.

Quickstart

pip install dataproc-spark-connect

from google.cloud.dataproc_spark_connect import DataprocSparkSession
spark = DataprocSparkSession.builder.projectId('my-project').location('us-central1').getOrCreate()
df = spark.createDataFrame([(1, 'data')], ['id', 'value'])
df.show()

Requires Google Cloud authentication credentials (GOOGLE_APPLICATION_CREDENTIALS environment variable or ADC) and appropriate IAM permissions for Dataproc Sessions and Session Templates.

Verify before relying

• Whether the package supports all Spark Connect features or has known limitations vs. native Spark Connect.
• Performance characteristics and latency when communicating with remote Dataproc sessions.
• Compatibility matrix with specific Dataproc runtime versions and Spark versions.

Package facts

Evidence: dataproc_spark_connect-1.1.0-py2.py3-none-any.whl

Tags

More Distributed Computing packages