dataengine

What it is and what it does

Dataengine is a configuration-driven data pipeline framework that wraps Apache Spark, pandas, and cloud service SDKs (AWS, Databricks, GitHub, Slack, Datadog) into a unified Python interface. The core abstraction is the Engine class, which orchestrates three main components: Database objects that represent data stores you connect to, Dataset objects that define data sources (local or S3), and Query objects that specify SQL transformations, input dependencies, and output destinations.

The package is designed to let you define complex data workflows declaratively through configuration files rather than imperative code. It sits in Alpha status and has not been updated in 484 days, suggesting either stable maintenance or dormancy. With 18 runtime dependencies including pyspark, pandas, boto3, and various cloud SDKs, it brings a large dependency footprint and assumes you have Spark and Java available.

Use it for:

• Define multi-step SQL transformations on Spark DataFrames with input/output dependencies managed through configuration.
• Orchestrate data loading from S3 or local sources, apply transformations, and write results back to databases or cloud storage.
• Integrate Slack notifications, GitHub metadata, or Datadog metrics into data pipeline workflows via SDK support.
• Manage multiple database connections (PostgreSQL, MySQL, Databricks) from a single Engine instance configured declaratively.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Dataengine is a Python framework for orchestrating data pipelines that integrates pandas, Apache Spark, and cloud services (AWS, Databricks, GitHub, Slack, Datadog) through a configuration-driven Engine class that manages Database, Dataset, and Query objects.

Yes, with conditions. Install if you need a lightweight configuration-driven wrapper around Spark SQL pipelines and have the Spark/Java runtime available. The low install friction and lack of known vulnerabilities are positives. However, the Alpha status, 484-day staleness, unclear license, and heavy dependency footprint (18 runtime deps) mean you should verify the license terms, confirm the configuration format works for your use case (the docs have a TODO), and assess whether the project's maintenance cadence suits your risk tolerance.

Install

dataengine on PyPI

pip install dataengine

uv add dataengine

poetry add dataengine

Installing dataengine

Before you install

Low install friction with a pure-Python wheel, but the package is in Alpha status and has not been updated in 484 days. It carries 18 runtime dependencies including heavy libraries like pyspark and pandas, which will pull in substantial transitive requirements.

License in practice

License status is unclear—no SPDX identifier or raw license text is available in the package metadata. Before adopting this in a commercial or regulated context, verify the actual license terms directly from the repository.

Quickstart

pip install dataengine

from dataengine import Engine

# Configure Engine with Database, Dataset, and Query subclasses
engine = Engine()
# (See repository for configuration file format)

Requires Python >=3.9. Apache Spark and its Java dependencies must be available in your environment for full functionality.

Verify before relying

• What configuration file format does Engine expect, and are there working examples beyond the TODO note in the description?
• Does the package support Python versions beyond 3.9, or is 3.9 the only tested version?
• What is the actual license of this package, and under what terms can it be used?

Package facts

Evidence: dataengine-0.0.92-py3-none-any.whl

Tags

More Software Development packages