intake

What it is and what it does

Intake is a data access abstraction layer that lets you describe datasets declaratively—via YAML or Python—and organize them into searchable catalogs. Instead of scattering data-loading logic throughout your code, you define data sources once and reference them by name. It handles the mechanics of connecting to remote storage (S3, GCS, etc.), parsing different formats, and optionally transforming data on load.

The package is built on fsspec for filesystem abstraction, pyyaml for configuration, networkx for dependency graphs, and platformdirs for configuration storage. It's designed for teams sharing datasets and for workflows where reproducibility and data provenance matter. You can search catalogs, load data into memory or stream it, and chain transformations—all without writing custom I/O code for each data source.

Use it for:

• Define a shared data catalog in YAML so team members load datasets by name instead of hardcoding paths or credentials.
• Discover and load datasets from a central repository without knowing their exact format or storage location.
• Build reproducible data pipelines by versioning and referencing datasets through a catalog rather than file paths.
• Integrate data from multiple remote sources (cloud storage, APIs, databases) under a unified interface.
• Automate data loading and transformation workflows for machine learning or scientific computing projects.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Intake provides a declarative data catalog system for describing, discovering, and loading datasets from multiple sources and formats, with support for remote storage and compute platforms.

Yes, if you work with multiple data sources or share datasets across a team. Intake reduces boilerplate and centralizes data access logic. However, note the active security advisory (GHSA-37g4-qqqv-7m99) and verify it does not affect your use case before deploying to production. The plugin ecosystem may require additional dependencies for your specific data sources.

Install

intake on PyPI

pip install intake

uv add intake

poetry add intake

Installing intake

Before you install

Low install friction with a pure-Python wheel and four runtime dependencies (fsspec, pyyaml, platformdirs, networkx). Maintenance status is active with a recent release.

License in practice

MIT license permits commercial and private use with minimal restrictions; you must include the license text in distributions.

Quickstart

pip install intake

import intake

# Load a data source from a catalog or direct specification
data = intake.open_csv('file.csv')

Requires Python 3.10 or later. Specific data drivers and plugins may require additional dependencies beyond the base install.

Verify before relying

• Whether GHSA-37g4-qqqv-7m99 affects your intended use case and whether a patch is available.
• Whether the package's plugin/driver ecosystem is mature enough for your data source types.
• Performance characteristics when working with large catalogs or remote datasets.

Package facts

Evidence: intake-2.0.9-py3-none-any.whl

Tags

More Database packages