{"categories":[{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing/3"}],"enrichment":{"capability":"Wraps Apache Spark Connect to let Python applications connect to remote Dataproc Spark sessions via the Spark Connect protocol without extra setup steps.","skillfed_tags":["google-cloud","spark-connect","dataproc"],"use_cases":["Run distributed Spark jobs on Dataproc from a local Python script or Jupyter notebook without manual session provisioning.","Share a single Spark session across multiple notebooks by reusing named session IDs, reducing startup time and costs.","Execute Spark SQL queries interactively in Jupyter using magic commands (with sparksql-magic installed).","Configure Spark executor memory, cores, and Dataproc runtime versions via a fluent builder API.","Manage session TTL and idle timeouts to control Dataproc resource lifecycle and billing."],"what_it_does":"This package is a Python client that simplifies connecting to remote Apache Spark sessions running on Google Cloud Dataproc using the Spark Connect protocol. It wraps the Spark Connect client with additional conveniences for Dataproc-specific session management, eliminating manual setup steps and providing a fluent builder API for configuration.\n\nYou use it to create or reuse named Spark sessions in Dataproc, configure Spark properties and session templates, and execute distributed computations from Python notebooks or scripts. It handles Google Cloud authentication and session lifecycle management, and optionally integrates with Jupyter magic commands for SQL queries. The package depends on pyspark, google-cloud-dataproc, google-api-core, packaging, tqdm, and websockets.","worth_installing":"Yes. The package is actively maintained, has low install friction, carries no known vulnerabilities, and is licensed permissively. It is worth installing if you need to run Spark workloads on Google Cloud Dataproc from Python and want to avoid manual session setup. The main prerequisite is Google Cloud authentication and appropriate IAM permissions."},"id":"dataproc-spark-connect","links":{"html":"https://skillfed.io/packages/dataproc-spark-connect","md":"https://skillfed.io/packages/dataproc-spark-connect.md","pypi":"https://pypi.org/project/dataproc-spark-connect/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-04-07","license_spdx":null,"license_treatment":"permissive","name":"dataproc-spark-connect","python_support":"unspecified","summary":"Dataproc client library for Spark Connect"},"popularity":{"monthly_downloads":95182,"position":13286,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"1.1.0"}