databricks-connect

What it is and what it does

Databricks Connect bridges your local development environment—IDE, notebook server, or custom application—to a remote Databricks cluster. Instead of installing and running Spark locally, you write code using familiar Spark APIs and execute it on a managed Databricks cluster, letting the cluster handle computation while your local machine remains lightweight.

The package depends on py4j for JVM interop, grpcio for remote communication, and standard data libraries like pandas, pyarrow, and numpy. It's actively maintained and has low install friction as a pure-Python wheel, but it locks you to Python 3.12 and requires a Databricks workspace with valid credentials to function.

Use it for:

• Develop and test Spark jobs in your IDE without installing a local Spark cluster.
• Run interactive Spark queries from a notebook server connected to a shared Databricks workspace.
• Integrate Spark computation into a Python application running on a lightweight client machine.
• Prototype data pipelines locally before deploying them to production Databricks clusters.
• Use Databricks-managed infrastructure for compute-heavy workloads while keeping your development loop fast.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Databricks Connect is a client library that lets you write Spark code locally in your IDE or notebook and execute it remotely on a Databricks cluster instead of running it locally.

Yes, if you have a Databricks workspace and need to run Spark code from a local IDE or notebook. The low install friction, active maintenance, and zero known vulnerabilities make it safe to adopt. The Python 3.12 requirement and proprietary license are the main constraints—verify the license terms for your use case and ensure your environment supports Python 3.12.

Install

databricks-connect on PyPI

pip install databricks-connect

uv add databricks-connect

poetry add databricks-connect

Installing databricks-connect

Before you install

Low install friction with a pure-Python wheel. Active maintenance with a release 14 days ago. Requires Python 3.12 specifically, which narrows compatibility to current-version environments only.

License in practice

Licensed under Databricks Proprietary License with unclear treatment. You should review the license terms directly before adopting in a commercial or redistributable context.

Quickstart

pip install databricks-connect==19.0.0

from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.build()
df = spark.sql('SELECT * FROM my_table')
df.show()

Requires Python 3.12 exactly; connection to a Databricks workspace with valid credentials configured (typically via DATABRICKS_HOST and DATABRICKS_TOKEN environment variables or Databricks CLI config).

Verify before relying

• Whether the proprietary license permits use in closed-source commercial applications or only in open-source contexts.
• Whether Python 3.12 requirement will be relaxed in future releases or remains pinned.
• Performance characteristics and latency overhead of remote Spark execution compared to local Spark.

Package facts

Evidence: databricks_connect-19.0.0-py2.py3-none-any.whl

Tags

More Distributed Computing packages