dlt

What it is and what it does

dlt is a Python library that handles the repetitive parts of data pipelines: extracting from REST APIs, SQL databases, cloud storage, or DataFrames; inferring and normalizing schemas automatically; and loading into any of 20+ destinations. You define sources declaratively using decorators or configuration objects, then point a pipeline at a destination—dlt manages credentials, DDL, type mapping, staging, and schema drift for you. It's a library, not a platform: you pip-install it into your existing code and keep your workflow intact.

The package supports incremental loading (load only new or changed rows), merge strategies (upsert on primary key), schema contracts (freeze, evolve, or discard unexpected data), and secrets injection from environment variables or config files. It works anywhere Python runs—Colab notebooks, AWS Lambda, Airflow DAGs, local scripts, or AI coding agents. The Dataset API lets you reconnect to a pipeline by name and read tables back in the format your tool needs (DataFrame, SQL query, etc.).

Use it for:

• Load REST API data into DuckDB or Snowflake with automatic pagination, filtering, and schema inference.
• Replicate tables from a MySQL or PostgreSQL database into a data warehouse with incremental updates.
• Ingest CSV or Parquet files from S3, GCS, or Azure into a destination, handling schema drift automatically.
• Build an Airflow DAG that extracts from multiple sources, normalizes nested data, and upserts into BigQuery.
• Merge pandas or Polars DataFrames into a warehouse with zero-copy Arrow support and type safety.
• Enforce data quality at the gate using schema contracts that reject or adapt unexpected columns and types.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

dlt automates extraction, schema inference, and loading of data from REST APIs, SQL databases, files, and DataFrames into 20+ destinations (DuckDB, Snowflake, BigQuery, Postgres, etc.) with declarative resource definitions and zero-copy Arrow support.

Yes. dlt is actively maintained, has no known vulnerabilities, low install friction, and a permissive license. It solves a real problem—automating tedious data loading—with a clean, Pythonic API and support for many sources and destinations. The declarative resource model and schema inference reduce boilerplate significantly. Start with a simple REST API or SQL database extraction to evaluate fit.

Install

dlt on PyPI

pip install dlt

uv add dlt

poetry add dlt

Installing dlt

Before you install

Low install friction with a pure-wheel distribution and 26 runtime dependencies already packaged. Active maintenance—last release 3 days ago, repository at 5738 stars, continuous commits. Supports Python 3.10 through 3.14, though 3.14 support is noted as experimental.

License in practice

Apache-2.0 permissive license allows commercial use, modification, and distribution with minimal restrictions. No licensing concerns for most use cases.

Quickstart

pip install dlt

import dlt
from dlt.sources.rest_api import rest_api_source

source = rest_api_source({
    "client": {"base_url": "https://pokeapi.co/api/v2/"},
    "resources": [{"name": "pokemon", "endpoint": {"path": "pokemon"}}],
})

pipeline = dlt.pipeline(pipeline_name="pokemon", destination="duckdb", dataset_name="pokemon_data")
print(pipeline.run(source))

Requires Python 3.10 or later. Optional extras (e.g., dlt[duckdb], dlt[bigquery]) needed for specific destinations.

Verify before relying

• Whether the 5000+ sources mentioned in the description are pre-built integrations or community-contributed templates.
• Performance characteristics and scalability limits for large datasets or high-frequency incremental loads.
• Maturity and stability of Python 3.14 experimental support in production environments.

Package facts

Evidence: dlt-1.30.0-py3-none-any.whl

Keywords: etl

Tags

More Libraries packages