vortex-data
Python bindings for Vortex, an Apache Arrow-compatible toolkit for working with compressed array data.
Decision gist · record as of 2026-08-14
Yes, if you work with large columnar datasets and need faster random access or scan performance than Parquet, or if you want to experiment with pluggable compression encodings in an Arrow-compatible format. The package is actively maintained, has no known vulnerabilities, and uses a permissive license. Install friction is moderate due to compiled wheels, but pre-built binaries are available for common platforms. Not necessary if you're already satisfied with Parquet or don't need the performance gains.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.11 or later; compiled wheels available for macOS (x86_64, arm64) and Linux (x86_64, aarch64).
- Medium install friction due to compiled wheels for multiple platforms (macOS x86_64/arm64, Linux x86_64/aarch64).
- Active maintenance with releases within the past week and 3138 repository stars.
License · maintenance · safety
permissive license (permissive) — Licensed under Apache License 2.0 (permissive). No restrictions on commercial use, modification, or distribution provided attribution and license terms are included.
last release 2026-08-07 (7 days) · last repo commit 2026-08-14 · 3,138 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 203,397 downloads/mo, #9,630 on PyPI
Alternatives
Verify before relying
pip install vortex-data
import vortex
# Use vortex with pyarrow arrays for columnar data operations- Specific performance benchmarks (100x faster random access, 10-20x faster scans) are claimed in the description but not independently verified in the fact sheet.
- File format stability claim (backwards compatible from 0.36.0 onwards) applies to the Rust format; Python API stability guarantees are not explicitly stated.
- Integration status with Iceberg and other listed systems (Spark, Polars, DuckDB) in the Python bindings specifically.
What it is and what it does
Vortex-data is a Python interface to Vortex, a next-generation columnar storage format designed for object-storage-backed data systems. It separates logical schema from physical encoding, allowing pluggable compression strategies (RLE, dictionary, and others) while maintaining zero-copy compatibility with Apache Arrow. The package lets you read, write, and manipulate Vortex files from Python, integrating with the broader Arrow ecosystem.
The format is optimized for random access and scan performance on wide tables with efficient metadata handling. It comes with built-in encodings compatible with Arrow's memory layout and supports cascading compression schemes. The file format itself is considered stable from version 0.36.0 onwards, though the Python library APIs may evolve. Dependencies include pyarrow, substrait, and typing-extensions.
Use it for
- Store and query large columnar datasets in object storage with faster random access than Parquet.
- Build data pipelines that need efficient compression without sacrificing read performance.
- Integrate Vortex files into Arrow-based analytics workflows (DataFusion, DuckDB, Pandas).
- Experiment with alternative encoding strategies for specialized data types or access patterns.
- Benchmark columnar formats in performance-critical data systems.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you work with large columnar datasets and need faster random access or scan performance than Parquet, or if you want to experiment with pluggable compression encodings in an Arrow-compatible format.
The package is actively maintained, has no known vulnerabilities, and uses a permissive license. Install friction is moderate due to compiled wheels, but pre-built binaries are available for common platforms. Not necessary if you're already satisfied with Parquet or don't need the performance gains.
Install
vortex-data on PyPI
Before you install
Medium install friction due to compiled wheels for multiple platforms (macOS x86_64/arm64, Linux x86_64/aarch64). Active maintenance with releases within the past week and 3138 repository stars. Requires Python 3.11 or later.
Requires Python 3.11 or later; compiled wheels available for macOS (x86_64, arm64) and Linux (x86_64, aarch64).
License in practice
Licensed under Apache License 2.0 (permissive). No restrictions on commercial use, modification, or distribution provided attribution and license terms are included.
Quickstart
pip install vortex-data
import vortex
# Use vortex with pyarrow arrays for columnar data operationsVerify before relying
- Specific performance benchmarks (100x faster random access, 10-20x faster scans) are claimed in the description but not independently verified in the fact sheet.
- File format stability claim (backwards compatible from 0.36.0 onwards) applies to the Rust format; Python API stability guarantees are not explicitly stated.
- Integration status with Iceberg and other listed systems (Spark, Polars, DuckDB) in the Python bindings specifically.
Package facts
| License | permissive license permissive |
| Python support | Supports the current Python release >=3.11 |
| Install friction | Medium. Platform-specific wheel |
| Runtime dependencies | 3 packagespyarrowsubstraittyping-extensions |
| Maintenance | Actively maintained 7 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 203,397 / month, #9,630 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 4 - BetaIntended Audience :: DevelopersIntended Audience :: Science/ResearchLicense :: OSI Approved :: Apache Software LicenseProgramming Language :: Python :: 3Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Topic :: DatabaseTopic :: File FormatsTopic :: Scientific/Engineering |
Evidence: vortex_data-0.84.0-cp311-abi3-macosx_10_12_x86_64.whl; vortex_data-0.84.0-cp311-abi3-macosx_11_0_arm64.whl; vortex_data-0.84.0-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; vortex_data-0.84.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “columnar file format python”
- vortex-dataVortex-data provides Python bindings to work with Vortex, a columnar…
- feather-formatFeather-format provides a Python interface to store and load pandas…
- pyorcPyORC reads and writes Apache ORC files using Python, wrapping the…
Give your agent the search over MCP, or paste the wish link into any chat.
More Scientific/Engineering packages
NumPy provides an N-dimensional array object and a comprehensive suite of mathematical, linear algebra, Fourier transform, and random number functions for scientific computing in Python.
pandas provides fast, flexible data structures (Series and DataFrame) for loading, cleaning, transforming, and analyzing labeled or relational data in Python.
scipy provides numerical algorithms for mathematics, science, and engineering—including optimization, integration, linear algebra, Fourier transforms, signal and image processing, and ODE solvers—built on numpy arrays.
scikit-learn provides a comprehensive Python library for supervised and unsupervised machine learning, including classification, regression, clustering, dimensionality reduction, and model evaluation tools built on NumPy and SciPy.
Install it if you need to train, evaluate, or deploy supervised or unsupervised learning models.
dill extends Python's pickle module to serialize and deserialize a much wider range of Python objects, including functions, lambdas, classes, and interpreter sessions, to byte streams for storage or network transmission.
Multiprocess is an enhanced fork of Python's standard multiprocessing library that uses dill for better serialization, allowing you to spawn processes with a threading-like API and share complex objects between them.
Install it if you use multiprocessing and encounter pickle serialization limits with lambdas or complex objects.
See also datafusion · pyarrow · pylance · feather-format · total-perspective-vortex · hepconvert · geoarrow-c · arrow-odbc · awkward0 · fastparquet