---
id: dataproc-spark-connect
version: "1.1.0"
license: Apache 2.0
license_treatment: permissive
maintenance: active
---

# dataproc-spark-connect — Dataproc client library for Spark Connect

License: permissive · Maintenance: active · Downloads: 95.2K/mo

## What it is and what it does

This package is a Python client that simplifies connecting to remote Apache Spark sessions running on Google Cloud Dataproc using the Spark Connect protocol. It wraps the Spark Connect client with additional conveniences for Dataproc-specific session management, eliminating manual setup steps and providing a fluent builder API for configuration.

You use it to create or reuse named Spark sessions in Dataproc, configure Spark properties and session templates, and execute distributed computations from Python notebooks or scripts. It handles Google Cloud authentication and session lifecycle management, and optionally integrates with Jupyter magic commands for SQL queries. The package depends on pyspark, google-cloud-dataproc, google-api-core, packaging, tqdm, and websockets.

Use it for:

- Run distributed Spark jobs on Dataproc from a local Python script or Jupyter notebook without manual session provisioning.
- Share a single Spark session across multiple notebooks by reusing named session IDs, reducing startup time and costs.
- Execute Spark SQL queries interactively in Jupyter using magic commands (with sparksql-magic installed).
- Configure Spark executor memory, cores, and Dataproc runtime versions via a fluent builder API.
- Manage session TTL and idle timeouts to control Dataproc resource lifecycle and billing.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Wraps Apache Spark Connect to let Python applications connect to remote Dataproc Spark sessions via the Spark Connect protocol without extra setup steps.

Yes. The package is actively maintained, has low install friction, carries no known vulnerabilities, and is licensed permissively. It is worth installing if you need to run Spark workloads on Google Cloud Dataproc from Python and want to avoid manual session setup. The main prerequisite is Google Cloud authentication and appropriate IAM permissions.

## Install

pip install dataproc-spark-connect
uv add dataproc-spark-connect
poetry add dataproc-spark-connect

## Installing dataproc-spark-connect

Before you install: Low install friction with a pure-Python wheel. Active maintenance as of 2026-04-07, with recent commits. Depends on google-cloud-dataproc, pyspark, and standard utilities; all are established packages.

License in practice: Licensed under Apache 2.0 (permissive), allowing use in commercial and private projects with minimal restrictions.

Quickstart:

pip install dataproc-spark-connect

from google.cloud.dataproc_spark_connect import DataprocSparkSession
spark = DataprocSparkSession.builder.projectId('my-project').location('us-central1').getOrCreate()
df = spark.createDataFrame([(1, 'data')], ['id', 'value'])
df.show()

Requires Google Cloud authentication credentials (GOOGLE_APPLICATION_CREDENTIALS environment variable or ADC) and appropriate IAM permissions for Dataproc Sessions and Session Templates.

Verify before relying:

- Whether the package supports all Spark Connect features or has known limitations vs. native Spark Connect.
- Performance characteristics and latency when communicating with remote Dataproc sessions.
- Compatibility matrix with specific Dataproc runtime versions and Spark versions.

## Package facts

- License: Apache 2.0 (permissive)
- Python support: unspecified
- Install friction: low
- Maintenance: active
- Downloads: 95.2K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

spark connect dataproc client, remote spark session google cloud, pyspark dataproc serverless, spark connect python wrapper, google cloud dataproc spark, distributed spark computing, spark session management, google-cloud, spark-connect, dataproc

[View on SkillFed](https://skillfed.io/packages/dataproc-spark-connect) · [View on PyPI](https://pypi.org/project/dataproc-spark-connect/)