pgpq

What it is and what it does

pgpq is a Python library that translates PyArrow RecordBatches into PostgreSQL's native binary wire format, enabling direct bulk loading of Arrow data into Postgres via the COPY command. It handles schema translation, field encoding, and binary serialization so you can stream Arrow datasets (from Parquet, CSV, or other sources) directly into Postgres without intermediate conversions or row-by-row inserts.

The package sits between PyArrow and psycopg, providing an ArrowToPostgresBinaryEncoder that takes an Arrow schema, optionally customizes field encoders (e.g., mapping string columns to JSONB), and produces the byte stream PostgreSQL expects. You define a temporary or permanent table matching the encoder's schema, then pipe the encoded batches through psycopg's COPY interface. It's designed for data warehouse and ETL workflows where bulk loading speed matters.

Use it for:

• Load Parquet files from S3 or local disk into PostgreSQL without staging or intermediate formats.
• Bulk import Arrow datasets with custom field encoding (e.g., string-to-JSONB) to match Postgres table schemas.
• Stream large datasets through psycopg's COPY interface while avoiding row-by-row inserts or temporary CSV files.
• Automate data pipelines that extract Arrow data and load it into Postgres for analytics or reporting.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Encodes PyArrow RecordBatches into PostgreSQL's native binary format for efficient bulk loading via COPY.

Yes, if you regularly load Arrow data into PostgreSQL and want to avoid row-by-row inserts or intermediate file formats. The package is actively maintained, has no known vulnerabilities, and solves a specific ETL bottleneck. Medium install friction (compiled wheels) is typical for performance-critical libraries. Not necessary if you use PostgreSQL's native tools or rarely bulk-load Arrow data.

Install

pgpq on PyPI

pip install pgpq

uv add pgpq

poetry add pgpq

Installing pgpq

Before you install

Medium install friction due to compiled wheels (cp39-abi3 binaries for multiple platforms). Active maintenance with recent commits and steady releases since early 2023.

License in practice

MIT license permits commercial and private use with minimal restrictions; no copyleft obligations.

Quickstart

pip install pgpq

from pgpq import ArrowToPostgresBinaryEncoder
import pyarrow.dataset as ds

dataset = ds.dataset('/path/to/parquet')
encoder = ArrowToPostgresBinaryEncoder(dataset.schema)
pg_schema = encoder.schema()
# Use encoder.write_header(), encoder.write_batch(batch), encoder.finish() with psycopg COPY

Requires Python 3.10 or later; PostgreSQL server and psycopg connection for actual data transfer.

Verify before relying

• Performance characteristics compared to alternative bulk-load methods or native PostgreSQL tooling.
• Support for all PostgreSQL data types or limitations in type mapping beyond the documented examples.

Package facts

Evidence: pgpq-0.11.1-cp39-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl; pgpq-0.11.1-cp39-abi3-macosx_10_12_x86_64.whl; pgpq-0.11.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; pgpq-0.11.1-cp39-abi3-manylinux_2_5_i686.manylinux1_i686.whl; pgpq-0.11.1-cp39-abi3-musllinux_1_2_i686.whl; pgpq-0.11.1-cp39-abi3-musllinux_1_2_x86_64.whl; pgpq-0.11.1-cp39-abi3-win32.whl; pgpq-0.11.1-cp39-abi3-win_amd64.whl

Tags

More Software Development packages