{"categories":[{"label":"Software Development","url":"https://skillfed.io/packages/category/software-development/14"}],"enrichment":{"capability":"Dataengine is a Python framework for orchestrating data pipelines that integrates pandas, Apache Spark, and cloud services (AWS, Databricks, GitHub, Slack, Datadog) through a configuration-driven Engine class that manages Database, Dataset, and Query objects.","skillfed_tags":["spark-etl","configuration-driven","cloud-data"],"use_cases":["Define multi-step SQL transformations on Spark DataFrames with input/output dependencies managed through configuration.","Orchestrate data loading from S3 or local sources, apply transformations, and write results back to databases or cloud storage.","Integrate Slack notifications, GitHub metadata, or Datadog metrics into data pipeline workflows via SDK support.","Manage multiple database connections (PostgreSQL, MySQL, Databricks) from a single Engine instance configured declaratively."],"what_it_does":"Dataengine is a configuration-driven data pipeline framework that wraps Apache Spark, pandas, and cloud service SDKs (AWS, Databricks, GitHub, Slack, Datadog) into a unified Python interface. The core abstraction is the Engine class, which orchestrates three main components: Database objects that represent data stores you connect to, Dataset objects that define data sources (local or S3), and Query objects that specify SQL transformations, input dependencies, and output destinations.\n\nThe package is designed to let you define complex data workflows declaratively through configuration files rather than imperative code. It sits in Alpha status and has not been updated in 484 days, suggesting either stable maintenance or dormancy. With 18 runtime dependencies including pyspark, pandas, boto3, and various cloud SDKs, it brings a large dependency footprint and assumes you have Spark and Java available.","worth_installing":"Yes, with conditions. Install if you need a lightweight configuration-driven wrapper around Spark SQL pipelines and have the Spark/Java runtime available. The low install friction and lack of known vulnerabilities are positives. However, the Alpha status, 484-day staleness, unclear license, and heavy dependency footprint (18 runtime deps) mean you should verify the license terms, confirm the configuration format works for your use case (the docs have a TODO), and assess whether the project's maintenance cadence suits your risk tolerance."},"id":"dataengine","links":{"html":"https://skillfed.io/packages/dataengine","md":"https://skillfed.io/packages/dataengine.md","pypi":"https://pypi.org/project/dataengine/"},"maintenance":{"status":"aging"},"meta":{"latest_release":"2025-04-17","license_spdx":null,"license_treatment":"unclear","name":"dataengine","python_support":"supports_current","summary":"General purpose data engineering python package."},"popularity":{"monthly_downloads":159859,"position":10687,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.0.92"}