---
id: vortex-data
version: "0.84.0"
license: unclear
license_treatment: permissive
maintenance: active
---

# vortex-data — Python bindings for Vortex, an Apache Arrow-compatible toolkit for working with compressed array data.

License: permissive · Maintenance: active · Downloads: 203.4K/mo

## What it is and what it does

Vortex-data is a Python interface to Vortex, a next-generation columnar storage format designed for object-storage-backed data systems. It separates logical schema from physical encoding, allowing pluggable compression strategies (RLE, dictionary, and others) while maintaining zero-copy compatibility with Apache Arrow. The package lets you read, write, and manipulate Vortex files from Python, integrating with the broader Arrow ecosystem.

The format is optimized for random access and scan performance on wide tables with efficient metadata handling. It comes with built-in encodings compatible with Arrow's memory layout and supports cascading compression schemes. The file format itself is considered stable from version 0.36.0 onwards, though the Python library APIs may evolve. Dependencies include pyarrow, substrait, and typing-extensions.

Use it for:

- Store and query large columnar datasets in object storage with faster random access than Parquet.
- Build data pipelines that need efficient compression without sacrificing read performance.
- Integrate Vortex files into Arrow-based analytics workflows (DataFusion, DuckDB, Pandas).
- Experiment with alternative encoding strategies for specialized data types or access patterns.
- Benchmark columnar formats in performance-critical data systems.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Vortex-data provides Python bindings to work with Vortex, a columnar file format optimized for high-performance data processing with Apache Arrow compatibility and pluggable compression encodings.

Yes, if you work with large columnar datasets and need faster random access or scan performance than Parquet, or if you want to experiment with pluggable compression encodings in an Arrow-compatible format. The package is actively maintained, has no known vulnerabilities, and uses a permissive license. Install friction is moderate due to compiled wheels, but pre-built binaries are available for common platforms. Not necessary if you're already satisfied with Parquet or don't need the performance gains.

## Install

pip install vortex-data
uv add vortex-data
poetry add vortex-data

## Installing vortex-data

Before you install: Medium install friction due to compiled wheels for multiple platforms (macOS x86_64/arm64, Linux x86_64/aarch64). Active maintenance with releases within the past week and 3138 repository stars. Requires Python 3.11 or later.

License in practice: Licensed under Apache License 2.0 (permissive). No restrictions on commercial use, modification, or distribution provided attribution and license terms are included.

Quickstart:

pip install vortex-data

import vortex
# Use vortex with pyarrow arrays for columnar data operations

Requires Python 3.11 or later; compiled wheels available for macOS (x86_64, arm64) and Linux (x86_64, aarch64).

Verify before relying:

- Specific performance benchmarks (100x faster random access, 10-20x faster scans) are claimed in the description but not independently verified in the fact sheet.
- File format stability claim (backwards compatible from 0.36.0 onwards) applies to the Rust format; Python API stability guarantees are not explicitly stated.
- Integration status with Iceberg and other listed systems (Spark, Polars, DuckDB) in the Python bindings specifically.

## Package facts

- License: not declared (permissive)
- Python support: supports_current
- Install friction: medium
- Maintenance: active
- Downloads: 203.4K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

columnar file format python, arrow-compatible data compression, high-performance data storage, vortex file format bindings, compressed array data toolkit, columnar-storage, arrow-compatible, compression

[View on SkillFed](https://skillfed.io/packages/vortex-data) · [View on PyPI](https://pypi.org/project/vortex-data/)