---
id: dataengine
version: "0.0.92"
license: unclear
license_treatment: unclear
maintenance: aging
---

# dataengine — General purpose data engineering python package.

License: unclear · Maintenance: aging · Downloads: 159.9K/mo

## What it is and what it does

Dataengine is a configuration-driven data pipeline framework that wraps Apache Spark, pandas, and cloud service SDKs (AWS, Databricks, GitHub, Slack, Datadog) into a unified Python interface. The core abstraction is the Engine class, which orchestrates three main components: Database objects that represent data stores you connect to, Dataset objects that define data sources (local or S3), and Query objects that specify SQL transformations, input dependencies, and output destinations.

The package is designed to let you define complex data workflows declaratively through configuration files rather than imperative code. It sits in Alpha status and has not been updated in 484 days, suggesting either stable maintenance or dormancy. With 18 runtime dependencies including pyspark, pandas, boto3, and various cloud SDKs, it brings a large dependency footprint and assumes you have Spark and Java available.

Use it for:

- Define multi-step SQL transformations on Spark DataFrames with input/output dependencies managed through configuration.
- Orchestrate data loading from S3 or local sources, apply transformations, and write results back to databases or cloud storage.
- Integrate Slack notifications, GitHub metadata, or Datadog metrics into data pipeline workflows via SDK support.
- Manage multiple database connections (PostgreSQL, MySQL, Databricks) from a single Engine instance configured declaratively.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Dataengine is a Python framework for orchestrating data pipelines that integrates pandas, Apache Spark, and cloud services (AWS, Databricks, GitHub, Slack, Datadog) through a configuration-driven Engine class that manages Database, Dataset, and Query objects.

Yes, with conditions. Install if you need a lightweight configuration-driven wrapper around Spark SQL pipelines and have the Spark/Java runtime available. The low install friction and lack of known vulnerabilities are positives. However, the Alpha status, 484-day staleness, unclear license, and heavy dependency footprint (18 runtime deps) mean you should verify the license terms, confirm the configuration format works for your use case (the docs have a TODO), and assess whether the project's maintenance cadence suits your risk tolerance.

## Install

pip install dataengine
uv add dataengine
poetry add dataengine

## Installing dataengine

Before you install: Low install friction with a pure-Python wheel, but the package is in Alpha status and has not been updated in 484 days. It carries 18 runtime dependencies including heavy libraries like pyspark and pandas, which will pull in substantial transitive requirements.

License in practice: License status is unclear—no SPDX identifier or raw license text is available in the package metadata. Before adopting this in a commercial or regulated context, verify the actual license terms directly from the repository.

Quickstart:

pip install dataengine

from dataengine import Engine

# Configure Engine with Database, Dataset, and Query subclasses
engine = Engine()
# (See repository for configuration file format)

Requires Python >=3.9. Apache Spark and its Java dependencies must be available in your environment for full functionality.

Verify before relying:

- What configuration file format does Engine expect, and are there working examples beyond the TODO note in the description?
- Does the package support Python versions beyond 3.9, or is 3.9 the only tested version?
- What is the actual license of this package, and under what terms can it be used?

## Package facts

- License: not declared (unclear)
- Python support: supports_current
- Install friction: low
- Maintenance: aging
- Downloads: 159.9K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

data pipeline orchestration, spark sql workflow framework, cloud data engineering, pandas spark integration, etl configuration management, distributed data processing, query execution framework, spark-etl, configuration-driven, cloud-data

[View on SkillFed](https://skillfed.io/packages/dataengine) · [View on PyPI](https://pypi.org/project/dataengine/)