acryl-executor

What it is and what it does

Acryl Executor is a remote execution agent for DataHub that isolates metadata ingestion tasks in dedicated subprocesses and virtual environments. It prevents dependency conflicts between different connectors by running each ingestion in its own isolated process, and integrates directly with the DataHub UI to execute ingestion jobs on demand via RUN_INGEST tasks. It also supports connection testing before full ingestion runs via TEST_CONNECTION tasks.

The package automatically compresses and uploads execution logs and artifacts to S3, with optional cleanup of local files after successful upload. It provides a plugin architecture for registering custom task implementations via Python entry points, and includes built-in secret store plugins for AWS Secrets Manager and GCP Secret Manager. Configuration is handled through environment variables for S3 bucket, path prefix, and cleanup behavior.

Use it for:

• Run DataHub metadata ingestion connectors in isolated subprocesses to prevent dependency version conflicts between different data source connectors.
• Trigger metadata ingestion jobs from the DataHub UI with automatic subprocess execution and cloud log storage.
• Validate connectivity to data sources before executing full ingestion runs using TEST_CONNECTION tasks.
• Store ingestion logs and artifacts in S3 with automatic compression and optional local cleanup after successful upload.
• Extend the executor with custom task implementations by registering plugins via Python entry points.
• Retrieve ingestion credentials from AWS Secrets Manager or GCP Secret Manager during task execution.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Acryl Executor runs DataHub metadata ingestion tasks in isolated subprocesses with S3 log storage, plugin extensibility, and UI-triggered workflow support.

Yes. Active maintenance, permissive Apache 2.0 license, low install friction, and no known vulnerabilities. Install if you are running DataHub and need remote subprocess isolation for metadata ingestion tasks, particularly when managing multiple connectors with conflicting dependencies or requiring S3 log storage. Not relevant if you are not using DataHub.

Install

acryl-executor on PyPI

pip install acryl-executor

uv add acryl-executor

poetry add acryl-executor

Installing acryl-executor

Before you install

Low install friction with a pure-Python wheel. Active maintenance (released 5 days ago). Requires Python 3.10+. Depends on acryl-datahub and standard libraries (pydantic, boto3, sqlalchemy-stubs, loguru); no compiled dependencies.

License in practice

Apache License 2.0 (permissive). You may use, modify, and distribute this package freely in commercial and private projects, provided you include a copy of the license and state significant changes.

Quickstart

pip install acryl-executor
python3 -m venv venv
source venv/bin/activate
pip install acryl-executor
# Configure S3 logging via environment variables:
# export DATAHUB_CLOUD_LOG_BUCKET=my-bucket
# export DATAHUB_CLOUD_LOG_PATH=/logs
# Then trigger ingestion tasks via DataHub UI or programmatically

Requires Python 3.10 or later. S3 log storage requires AWS credentials and boto3 configuration. Designed to run as a remote agent, typically deployed alongside DataHub.

Verify before relying

• Whether the package supports running on Windows (classifiers list POSIX/Linux/Unix only, but Python 3.10+ support suggests potential compatibility).
• Performance characteristics when handling large ingestion workloads or many concurrent subprocess tasks.
• Whether custom task plugins can be loaded from external packages or only from the local environment.

Package facts

Evidence: acryl_executor-0.3.19-py3-none-any.whl

Tags

More Software Development packages