vortex-data

What it is and what it does

Vortex-data is a Python interface to Vortex, a next-generation columnar storage format designed for object-storage-backed data systems. It separates logical schema from physical encoding, allowing pluggable compression strategies (RLE, dictionary, and others) while maintaining zero-copy compatibility with Apache Arrow. The package lets you read, write, and manipulate Vortex files from Python, integrating with the broader Arrow ecosystem.

The format is optimized for random access and scan performance on wide tables with efficient metadata handling. It comes with built-in encodings compatible with Arrow's memory layout and supports cascading compression schemes. The file format itself is considered stable from version 0.36.0 onwards, though the Python library APIs may evolve. Dependencies include pyarrow, substrait, and typing-extensions.

Use it for:

• Store and query large columnar datasets in object storage with faster random access than Parquet.
• Build data pipelines that need efficient compression without sacrificing read performance.
• Integrate Vortex files into Arrow-based analytics workflows (DataFusion, DuckDB, Pandas).
• Experiment with alternative encoding strategies for specialized data types or access patterns.
• Benchmark columnar formats in performance-critical data systems.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Vortex-data provides Python bindings to work with Vortex, a columnar file format optimized for high-performance data processing with Apache Arrow compatibility and pluggable compression encodings.

Yes, if you work with large columnar datasets and need faster random access or scan performance than Parquet, or if you want to experiment with pluggable compression encodings in an Arrow-compatible format. The package is actively maintained, has no known vulnerabilities, and uses a permissive license. Install friction is moderate due to compiled wheels, but pre-built binaries are available for common platforms. Not necessary if you're already satisfied with Parquet or don't need the performance gains.

Install

vortex-data on PyPI

pip install vortex-data

uv add vortex-data

poetry add vortex-data

Installing vortex-data

Before you install

Medium install friction due to compiled wheels for multiple platforms (macOS x86_64/arm64, Linux x86_64/aarch64). Active maintenance with releases within the past week and 3138 repository stars. Requires Python 3.11 or later.

License in practice

Licensed under Apache License 2.0 (permissive). No restrictions on commercial use, modification, or distribution provided attribution and license terms are included.

Quickstart

pip install vortex-data

import vortex
# Use vortex with pyarrow arrays for columnar data operations

Requires Python 3.11 or later; compiled wheels available for macOS (x86_64, arm64) and Linux (x86_64, aarch64).

Verify before relying

• Specific performance benchmarks (100x faster random access, 10-20x faster scans) are claimed in the description but not independently verified in the fact sheet.
• File format stability claim (backwards compatible from 0.36.0 onwards) applies to the Rust format; Python API stability guarantees are not explicitly stated.
• Integration status with Iceberg and other listed systems (Spark, Polars, DuckDB) in the Python bindings specifically.

Package facts

Evidence: vortex_data-0.84.0-cp311-abi3-macosx_10_12_x86_64.whl; vortex_data-0.84.0-cp311-abi3-macosx_11_0_arm64.whl; vortex_data-0.84.0-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; vortex_data-0.84.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Tags

More Scientific/Engineering packages