databricks-test

What it is and what it does

databricks_test is a unit testing framework designed to let you test Databricks notebooks locally using pytest, without needing to run them on a Databricks cluster. It works by injecting mocked versions of Databricks-specific objects—spark, dbutils, display—into your notebook code when run locally, while those objects remain untouched when the notebook runs in Databricks itself. The framework supports PySpark operations including UDFs and Pandas UDFs, and lets you mock connectors to external systems like Azure Storage, S3, and SQL Data Warehouse.

The typical workflow is to add an instrumentation cell at the top of your notebook that conditionally loads databricks_test only when running locally (detected by checking if dbutils exists), then write test cases that instantiate a databricks_test session, configure mocks for external dependencies, run the notebook, and assert on the results. This allows you to validate ETL logic, data transformations, and notebook behavior in a fast, local test environment before deploying to Databricks.

Use it for:

• Test ETL notebooks that read CSV and write Parquet locally before running on a cluster.
• Mock widget inputs and dbutils calls to test parameterized notebook logic without Databricks.
• Validate PySpark transformations and UDFs in isolation using local temporary files.
• Mock external connectors (Azure Storage, S3, SQL DW) to test data pipeline integration points.
• Run pytest on notebook code as part of a CI/CD pipeline in a code repository.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides a unit testing framework for Databricks notebooks, allowing you to run notebook code locally with mocked Databricks objects like spark, dbutils, and display for pytest-based testing.

No. The project is archived and abandoned since 2021-01-26 with no maintenance since then. While it has low install friction and permissive licensing, the lack of updates means it is unlikely to work reliably with current Databricks APIs, PySpark versions, or modern Python environments. Only consider it if you are locked into an older Databricks setup that has not changed since 2020.

Install

databricks-test on PyPI

pip install databricks-test

uv add databricks-test

poetry add databricks-test

Installing databricks-test

Before you install

Installation is straightforward with no runtime dependencies, but the project is archived and abandoned since 2021-01-26 with no updates since then. Use only if your Databricks setup and notebook patterns remain compatible with that era.

License in practice

Licensed under MIT (permissive), so you can use and modify freely, but the lack of active maintenance means no updates to licensing terms or legal review.

Quickstart

pip install databricks_test

# In your notebook:
if 'dbutils' not in locals():
    import databricks_test
    databricks_test.inject_variables()

# In your test file:
import databricks_test

def test_notebook():
    with databricks_test.session() as dbrickstest:
        dbrickstest.run_notebook(".", "notebook_name")
        # assertions here

Requires pytest setup in your code project outside Databricks; only supports .py notebook format, not .ipynb or .dbc.

Verify before relying

• Whether the package works with current Databricks notebook APIs and PySpark versions given the last release was 2020-03-04.
• Compatibility with modern Python versions beyond what the classifier 'Programming Language :: Python :: 3' indicates.
• Whether mocking of newer Databricks extensions and connectors is supported.

Package facts

Evidence: databricks_test-0.0.4-py3-none-any.whl

Tags

More Testing packages