connectorx

What it is and what it does

ConnectorX is a Rust-based database connector that executes SQL queries and streams results directly into Python data structures (Pandas, PyArrow, Polars, Modin, Dask) with minimal memory overhead. It follows a zero-copy architecture, meaning data moves exactly once from the database to your destination, avoiding the multiple intermediate copies that traditional Python connectors perform. The library supports Postgres, MySQL, MariaDB, SQLite, Redshift, Clickhouse, SQL Server, Azure SQL Database, Oracle, Big Query, and Trino.

You use it by calling `cx.read_sql()` with a connection string and SQL query. For large tables, you can specify a partition column and partition count to split the query across multiple threads, each loading its partition in parallel. The library automatically determines the partition range, allocates memory based on row counts, and streams data row-wise or column-wise depending on the source. No runtime dependencies means installation is a single pip command on supported platforms.

Use it for:

• Load large tables (gigabytes+) from production databases into Pandas or Polars for analysis without exhausting system memory.
• Parallelize SQL query execution by partitioning on a numerical column to reduce total load time on multi-core systems.
• Replace pandas.read_sql() in existing data pipelines to cut memory usage and execution time without changing application logic.
• Join tables across two or more databases in a single query (experimental federated mode) without staging intermediate results.
• Stream query results into PyArrow or Polars for downstream ML or analytics workflows that expect columnar data.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

ConnectorX loads data from databases directly into Python dataframes (Pandas, PyArrow, Polars, Modin, Dask) using a Rust backend optimized for speed and memory efficiency, with optional parallel loading via partitioning.

Yes. ConnectorX is actively maintained, has no known vulnerabilities, supports current Python versions (3.10+), and offers measurable performance and memory gains over standard SQL connectors—especially for large datasets. The permissive MIT license and zero runtime dependencies make adoption low-risk. Install it if you regularly load data from databases; the one-line API and optional parallelism justify the medium wheel size.

Install

connectorx on PyPI

pip install connectorx

uv add connectorx

poetry add connectorx

Installing connectorx

Before you install

Medium install friction: precompiled wheels available for Python 3.10–3.13 on macOS (Intel and ARM), Linux (x86_64 and aarch64), and Windows, but no runtime dependencies means no secondary build burden once the wheel installs.

License in practice

MIT license is permissive; you can use, modify, and distribute ConnectorX freely in commercial and private projects with minimal restrictions.

Quickstart

pip install connectorx

import connectorx as cx

# Single-threaded load
df = cx.read_sql("postgresql://user:pass@localhost/mydb", "SELECT * FROM table")

# Parallel load with partitioning
df = cx.read_sql(
    "postgresql://user:pass@localhost/mydb",
    "SELECT * FROM table",
    partition_on="id",
    partition_num=4
)

Requires Python 3.10 or later; precompiled wheels available for macOS, Linux, and Windows—building from source requires Rust toolchain.

Verify before relying

• Whether federated query support (joining across multiple databases) is production-ready or remains experimental.
• Specific performance gains on your hardware and database size relative to the 10x TPC-H lineitem benchmark cited.

Package facts

Evidence: connectorx-0.4.5-cp310-cp310-macosx_10_7_x86_64.whl; connectorx-0.4.5-cp310-cp310-macosx_11_0_arm64.whl; connectorx-0.4.5-cp310-cp310-manylinux_2_28_aarch64.whl; connectorx-0.4.5-cp310-cp310-manylinux_2_28_x86_64.whl; connectorx-0.4.5-cp310-none-win_amd64.whl; connectorx-0.4.5-cp311-cp311-macosx_10_7_x86_64.whl; connectorx-0.4.5-cp311-cp311-macosx_11_0_arm64.whl; connectorx-0.4.5-cp311-cp311-manylinux_2_28_aarch64.whl; connectorx-0.4.5-cp311-cp311-manylinux_2_28_x86_64.whl; connectorx-0.4.5-cp311-none-win_amd64.whl; connectorx-0.4.5-cp312-cp312-macosx_10_7_x86_64.whl; connectorx-0.4.5-cp312-cp312-macosx_11_0_arm64.whl; connectorx-0.4.5-cp312-cp312-manylinux_2_28_aarch64.whl; connectorx-0.4.5-cp312-cp312-manylinux_2_28_x86_64.whl; connectorx-0.4.5-cp312-none-win_amd64.whl; connectorx-0.4.5-cp313-cp313-macosx_10_7_x86_64.whl; connectorx-0.4.5-cp313-cp313-macosx_11_0_arm64.whl; connectorx-0.4.5-cp313-cp313-manylinux_2_28_aarch64.whl; connectorx-0.4.5-cp313-cp313-manylinux_2_28_x86_64.whl; connectorx-0.4.5-cp313-none-win_amd64.whl

Tags

More Database packages