kedro-datasets

What it is and what it does

Kedro-Datasets is a plugin that extends Kedro's DataCatalog with a collection of AbstractDataset implementations, allowing you to read and write data in many formats (CSV, Excel, Parquet, JSON, HDF5, SQL, Spark, images, and more) across different storage backends (local, network, cloud object stores, Hadoop). It's organized into groups by data type—pandas, spark, networkx, matplotlib, yaml—so you can install only the dependencies you need for your specific workflow.

The package acts as a bridge between Kedro's data abstraction layer and the actual storage and format libraries. Instead of writing custom dataset classes for each file type and storage combination, you declare your data sources in Kedro's configuration and let kedro-datasets handle the loading and saving. It supports optional group-level and dataset-level dependency installation, so a project using only CSV files doesn't need Spark or database drivers.

Use it for:

• Load CSV or Parquet files into pandas DataFrames within a Kedro pipeline without writing custom dataset code.
• Read and write Spark DataFrames to cloud object stores (S3, GCS) through Kedro's DataCatalog.
• Query SQL databases and cache results as datasets in a reproducible data pipeline.
• Work with multiple data formats (Excel, JSON, HDF5) in a single Kedro project with consistent APIs.
• Extend Kedro pipelines to handle image data or custom formats by implementing your own AbstractDataset.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Kedro-Datasets provides data connectors for Kedro's DataCatalog, implementing AbstractDataset for formats like CSV, Excel, Parquet, JSON, SQL, and Spark DataFrames across local, network, and cloud storage.

Yes. Kedro-Datasets is actively maintained, has low install friction, carries no known vulnerabilities, and is essential if you're using Kedro for data pipelines. It provides the standard data connectors most projects need and is permissively licensed. Install it as part of your Kedro setup and add optional group dependencies only for the formats your pipeline actually uses.

Install

kedro-datasets on PyPI

pip install kedro-datasets

uv add kedro-datasets

poetry add kedro-datasets

Installing kedro-datasets

Before you install

Low friction install with only three runtime dependencies (kedro, backports.strenum, lazy_loader). Active maintenance with releases every 7 days and a recent commit history.

License in practice

Licensed under Apache 2.0 (permissive), allowing commercial use and modification with attribution.

Quickstart

pip install kedro-datasets

from kedro_datasets.pandas import CSVDataset

ds = CSVDataset(filepath='data.csv')
df = ds.load()

Requires Python 3.10 or later; optional group-level dependencies (e.g., pandas, spark) must be installed separately for specific dataset types.

Verify before relying

• Whether lazy_loader is used for runtime performance optimization or just module discovery.
• Specific performance characteristics when working with large files or cloud storage.
• Whether all dataset types work equally well across local, network, and cloud backends.

Package facts

Evidence: kedro_datasets-9.6.0-py3-none-any.whl

Tags

More Database packages