---
id: pgpq
version: "0.11.1"
license: MIT
license_treatment: permissive
maintenance: active
---

# pgpq — Arrow -> PostgreSQL encoder

License: permissive · Maintenance: active · Downloads: 173.3K/mo

## What it is and what it does

pgpq is a Python library that translates PyArrow RecordBatches into PostgreSQL's native binary wire format, enabling direct bulk loading of Arrow data into Postgres via the COPY command. It handles schema translation, field encoding, and binary serialization so you can stream Arrow datasets (from Parquet, CSV, or other sources) directly into Postgres without intermediate conversions or row-by-row inserts.

The package sits between PyArrow and psycopg, providing an ArrowToPostgresBinaryEncoder that takes an Arrow schema, optionally customizes field encoders (e.g., mapping string columns to JSONB), and produces the byte stream PostgreSQL expects. You define a temporary or permanent table matching the encoder's schema, then pipe the encoded batches through psycopg's COPY interface. It's designed for data warehouse and ETL workflows where bulk loading speed matters.

Use it for:

- Load Parquet files from S3 or local disk into PostgreSQL without staging or intermediate formats.
- Bulk import Arrow datasets with custom field encoding (e.g., string-to-JSONB) to match Postgres table schemas.
- Stream large datasets through psycopg's COPY interface while avoiding row-by-row inserts or temporary CSV files.
- Automate data pipelines that extract Arrow data and load it into Postgres for analytics or reporting.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Encodes PyArrow RecordBatches into PostgreSQL's native binary format for efficient bulk loading via COPY.

Yes, if you regularly load Arrow data into PostgreSQL and want to avoid row-by-row inserts or intermediate file formats. The package is actively maintained, has no known vulnerabilities, and solves a specific ETL bottleneck. Medium install friction (compiled wheels) is typical for performance-critical libraries. Not necessary if you use PostgreSQL's native tools or rarely bulk-load Arrow data.

## Install

pip install pgpq
uv add pgpq
poetry add pgpq

## Installing pgpq

Before you install: Medium install friction due to compiled wheels (cp39-abi3 binaries for multiple platforms). Active maintenance with recent commits and steady releases since early 2023.

License in practice: MIT license permits commercial and private use with minimal restrictions; no copyleft obligations.

Quickstart:

pip install pgpq

from pgpq import ArrowToPostgresBinaryEncoder
import pyarrow.dataset as ds

dataset = ds.dataset('/path/to/parquet')
encoder = ArrowToPostgresBinaryEncoder(dataset.schema)
pg_schema = encoder.schema()
# Use encoder.write_header(), encoder.write_batch(batch), encoder.finish() with psycopg COPY

Requires Python 3.10 or later; PostgreSQL server and psycopg connection for actual data transfer.

Verify before relying:

- Performance characteristics compared to alternative bulk-load methods or native PostgreSQL tooling.
- Support for all PostgreSQL data types or limitations in type mapping beyond the documented examples.

## Package facts

- License: MIT (permissive)
- Python support: supports_current
- Install friction: medium
- Maintenance: active
- Downloads: 173.3K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

arrow to postgres binary encoder, bulk load arrow data to postgresql, pyarrow postgres copy format, arrow recordbatch postgres, efficient postgres data import, etl, bulk-load, data-pipeline

[View on SkillFed](https://skillfed.io/packages/pgpq) · [View on PyPI](https://pypi.org/project/pgpq/)