acryl-executor
Run DataHub metadata ingestion tasks remotely via subprocess isolation with S3 log storage
Decision gist · record as of 2026-08-14
Yes. Active maintenance, permissive Apache 2.0 license, low install friction, and no known vulnerabilities. Install if you are running DataHub and need remote subprocess isolation for metadata ingestion tasks, particularly when managing multiple connectors with conflicting dependencies or requiring S3 log storage. Not relevant if you are not using DataHub.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.10 or later.
- S3 log storage requires AWS credentials and boto3 configuration.
- Designed to run as a remote agent, typically deployed alongside DataHub.
License · maintenance · safety
Apache License 2.0 (permissive) — Apache License 2.0 (permissive). You may use, modify, and distribute this package freely in commercial and private projects, provided you include a copy of the license and state significant changes.
last release 2026-08-09 (5 days)
0 known vulnerabilities (OSV.dev, 2026-08-14) · 162,810 downloads/mo, #10,586 on PyPI
Alternatives
Verify before relying
pip install acryl-executor
python3 -m venv venv
source venv/bin/activate
pip install acryl-executor
# Configure S3 logging via environment variables:
# export DATAHUB_CLOUD_LOG_BUCKET=my-bucket
# export DATAHUB_CLOUD_LOG_PATH=/logs
# Then trigger ingestion tasks via DataHub UI or programmatically- Whether the package supports running on Windows (classifiers list POSIX/Linux/Unix only, but Python 3.10+ support suggests potential compatibility).
- Performance characteristics when handling large ingestion workloads or many concurrent subprocess tasks.
- Whether custom task plugins can be loaded from external packages or only from the local environment.
What it is and what it does
Acryl Executor is a remote execution agent for DataHub that isolates metadata ingestion tasks in dedicated subprocesses and virtual environments. It prevents dependency conflicts between different connectors by running each ingestion in its own isolated process, and integrates directly with the DataHub UI to execute ingestion jobs on demand via RUN_INGEST tasks. It also supports connection testing before full ingestion runs via TEST_CONNECTION tasks.
The package automatically compresses and uploads execution logs and artifacts to S3, with optional cleanup of local files after successful upload. It provides a plugin architecture for registering custom task implementations via Python entry points, and includes built-in secret store plugins for AWS Secrets Manager and GCP Secret Manager. Configuration is handled through environment variables for S3 bucket, path prefix, and cleanup behavior.
Use it for
- Run DataHub metadata ingestion connectors in isolated subprocesses to prevent dependency version conflicts between different data source connectors.
- Trigger metadata ingestion jobs from the DataHub UI with automatic subprocess execution and cloud log storage.
- Validate connectivity to data sources before executing full ingestion runs using TEST_CONNECTION tasks.
- Store ingestion logs and artifacts in S3 with automatic compression and optional local cleanup after successful upload.
- Extend the executor with custom task implementations by registering plugins via Python entry points.
- Retrieve ingestion credentials from AWS Secrets Manager or GCP Secret Manager during task execution.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
Active maintenance, permissive Apache 2.0 license, low install friction, and no known vulnerabilities. Install if you are running DataHub and need remote subprocess isolation for metadata ingestion tasks, particularly when managing multiple connectors with conflicting dependencies or requiring S3 log storage. Not relevant if you are not using DataHub.
Install
acryl-executor on PyPI
Before you install
Low install friction with a pure-Python wheel. Active maintenance (released 5 days ago). Requires Python 3.10+. Depends on acryl-datahub and standard libraries (pydantic, boto3, sqlalchemy-stubs, loguru); no compiled dependencies.
Requires Python 3.10 or later. S3 log storage requires AWS credentials and boto3 configuration. Designed to run as a remote agent, typically deployed alongside DataHub.
License in practice
Apache License 2.0 (permissive). You may use, modify, and distribute this package freely in commercial and private projects, provided you include a copy of the license and state significant changes.
Quickstart
pip install acryl-executor
python3 -m venv venv
source venv/bin/activate
pip install acryl-executor
# Configure S3 logging via environment variables:
# export DATAHUB_CLOUD_LOG_BUCKET=my-bucket
# export DATAHUB_CLOUD_LOG_PATH=/logs
# Then trigger ingestion tasks via DataHub UI or programmaticallyVerify before relying
- Whether the package supports running on Windows (classifiers list POSIX/Linux/Unix only, but Python 3.10+ support suggests potential compatibility).
- Performance characteristics when handling large ingestion workloads or many concurrent subprocess tasks.
- Whether custom task plugins can be loaded from external packages or only from the local environment.
Package facts
| License | Apache License 2.0 permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 11 packagespydanticurllib3mypy_extensionsbotocoreloguruexpandvarsacryl-datahubtyping_extensionsboto3sqlalchemy-stubsanyio |
| Maintenance | Actively maintained 5 days since the last release |
| First released | |
| Downloads | 162,810 / month, #10,586 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableEnvironment :: ConsoleEnvironment :: MacOS XIntended Audience :: DevelopersIntended Audience :: Information TechnologyIntended Audience :: System AdministratorsLicense :: OSI ApprovedLicense :: OSI Approved :: Apache Software LicenseOperating System :: POSIX :: LinuxOperating System :: UnixProgramming Language :: PythonProgramming Language :: Python :: 3Programming Language :: Python :: 3 :: OnlyProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Topic :: Software Development |
Evidence: acryl_executor-0.3.19-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “datahub ingestion remote execution”
- acryl-executorAcryl Executor runs DataHub metadata ingestion tasks in isolated…
- acryl-datahubDataHub's ingestion framework and CLI for pulling metadata from 50+…
- acryl-datahub-airflow-pluginAutomatically captures pipeline lineage, task metadata, and run…
Give your agent the search over MCP, or paste the wish link into any chat.
More Software Development packages
Provides backported and experimental type hints for Python 3.9+, allowing use of newer typing features on older Python versions and enabling early experimentation with type system PEPs before they enter the standard library.
NumPy provides an N-dimensional array object and a comprehensive suite of mathematical, linear algebra, Fourier transform, and random number functions for scientific computing in Python.
FastAPI is a Python web framework for building REST APIs using type hints, with automatic request validation, serialization, and interactive API documentation.
Provides a way to document function parameters, class attributes, return types, and variables inline using Python's `Annotated` type hint syntax instead of traditional docstrings.
Typer builds command-line applications from Python functions using type hints, automatically generating help text, argument parsing, and shell completion.
Install it if you are building CLIs in Python.
Distlib provides low-level packaging utilities for building, distributing, and managing Python software—including metadata handling, version specifiers, wheel support, script installation, and dependency resolution.
See also acryl-datahub · acryl-datahub-airflow-plugin · datahub · acryl-datahub-actions · acryl-datahub-dagster-plugin · azure-monitor-ingestion · acryl-datahub-classify · openmetadata-ingestion · s3torchconnectorclient · spotinst-agent-2