dbldatagen

What it is and what it does

dbldatagen is a PySpark library for generating synthetic data within Databricks environments. It lets you define data generation specifications in code to produce repeatable, predictable datasets at scale—up to billions of rows—for testing, benchmarking, and demos. You can generate data conforming to existing schemas or create ad-hoc datasets, with support for all Spark SQL primitive types, date/timestamp ranges, discrete values, weighted distributions, arrays, and SQL expressions. It includes a plugin mechanism for third-party libraries and integrates with Delta Live Tables pipelines.

The library has no dependencies on packages outside the Databricks runtime, making it lightweight to install and use. It supports generating data with consistency between primary and foreign keys for join and merge scenarios, and can produce code from existing schemas. You can access generated data from Scala, R, and other languages via views, and persist or export results to external storage.

Use it for:

• Generate large test datasets for validating ETL pipelines and data transformations in Databricks without exposing real data.
• Create repeatable synthetic data for demos and benchmarking to measure cluster performance under controlled, reproducible conditions.
• Produce consistent multi-table datasets with matching foreign keys for testing join, merge, and Change Data Capture scenarios.
• Build feature arrays for machine learning model testing using weighted distributions and custom column expressions.
• Populate Delta Live Tables pipelines with synthetic data sources for end-to-end pipeline validation.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Generates synthetic data at scale within Databricks using Spark, supporting repeatable, predictable datasets for testing, benchmarking, and demos across all Spark SQL primitive types.

Yes, if you work within Databricks and need synthetic data generation. The library is actively maintained, has no external dependencies, and integrates seamlessly with Spark. However, verify the 'Databricks License' terms for your use case before installing, particularly if you plan to use it outside Databricks or in proprietary applications.

Install

dbldatagen on PyPI

pip install dbldatagen

uv add dbldatagen

poetry add dbldatagen

Installing dbldatagen

Before you install

Low install friction with no runtime dependencies outside Databricks. Actively maintained with recent commits; last release 749 days ago indicates a stable, mature project rather than rapid iteration.

License in practice

License treatment is unclear—listed as 'Databricks License' with no SPDX identifier. Review the actual license terms before use in proprietary or redistributed contexts.

Quickstart

import dbldatagen as dg
df = dg.Datasets(spark, "basic/user").get(rows=1000_000).build()
num_rows = df.count()

Requires PySpark 3.1.2 and Python 3.8 or later; designed for Databricks runtime 10.4 LTS and later. Databricks runtime 13.2 or later recommended for full Unity Catalog support.

Verify before relying

• Whether the 'Databricks License' permits use outside Databricks environments or in closed-source applications.
• Performance characteristics and scalability limits when generating billions of rows on specific cluster configurations.
• Plugin mechanism details and compatibility with third-party libraries like Faker beyond what the description states.

Package facts

Evidence: dbldatagen-0.4.0.post1-py3-none-any.whl

Tags

More Testing packages