datafusion

What it is and what it does

DataFusion is a Python library that wraps Apache Arrow's in-memory query engine, allowing you to execute SQL queries and DataFrame operations against structured data files (Parquet, CSV, JSON) without loading everything into memory at once. It includes a query optimizer that rewrites logical plans for efficiency and supports user-defined Python functions (UDFs and UDAFs) embedded in SQL queries. The library is designed as a foundation for building data systems in Python—projects like Dask SQL and distributed engines (Ballista, Ray) use its Python bindings for SQL parsing and query planning.

You work with DataFusion by creating a SessionContext, registering data sources (files or in-memory DataFrames), and then executing queries either as SQL strings or using a DataFrame API. Results can be collected as PyArrow batches, converted to Pandas DataFrames, or serialized in Substrait format. The package depends on cloudpickle, pyarrow, and typing-extensions, and is actively maintained with prebuilt wheels for common platforms.

Use it for:

• Query large Parquet files with SQL without loading the entire dataset into memory
• Build a SQL interface over CSV or JSON data sources for exploratory analysis
• Register Python UDFs in SQL queries to apply custom logic during query execution
• Convert query results to Pandas DataFrames for visualization or further analysis
• Serialize and deserialize query plans in Substrait format for cross-system compatibility

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

DataFusion is a Python binding to Apache Arrow's in-memory query engine, enabling SQL and DataFrame-based queries against CSV, Parquet, and JSON data with built-in query optimization.

Yes. DataFusion is actively maintained, has no known vulnerabilities, and offers a practical way to execute optimized SQL queries against structured data files in Python. Install it if you need SQL query capabilities over Parquet/CSV/JSON without building a custom query engine or loading entire datasets into memory. The medium install friction is offset by prebuilt wheels and straightforward setup.

Install

datafusion on PyPI

pip install datafusion

uv add datafusion

poetry add datafusion

Installing datafusion

Before you install

Medium install friction due to compiled wheels; prebuilt binaries available for macOS (x86_64, arm64), Linux (x86_64, aarch64), and Windows (amd64) across Python 3.10–3.14. Active maintenance with recent releases.

License in practice

Licensed under Apache License 2.0 (permissive). Safe for commercial and proprietary use with standard attribution requirements.

Quickstart

pip install datafusion

from datafusion import SessionContext

ctx = SessionContext()
ctx.register_parquet('taxi', 'data.parquet')
df = ctx.sql('SELECT * FROM taxi WHERE passenger_count > 1')
results = df.collect()

Requires Python 3.10 or later. Parquet/CSV/JSON files must be accessible at query time.

Verify before relying

• Performance characteristics and typical query latency compared to alternatives
• Memory overhead and scalability limits for large datasets
• Completeness of SQL dialect support and known limitations

Package facts

Evidence: datafusion-54.0.0-cp310-abi3-macosx_10_12_x86_64.whl; datafusion-54.0.0-cp310-abi3-macosx_11_0_arm64.whl; datafusion-54.0.0-cp310-abi3-manylinux_2_28_aarch64.whl; datafusion-54.0.0-cp310-abi3-manylinux_2_28_x86_64.whl; datafusion-54.0.0-cp310-abi3-win_amd64.whl; datafusion-54.0.0-cp314-cp314t-macosx_10_12_x86_64.whl; datafusion-54.0.0-cp314-cp314t-macosx_11_0_arm64.whl; datafusion-54.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl; datafusion-54.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl; datafusion-54.0.0-cp314-cp314t-win_amd64.whl

Keywords: dataframe, datafusion, query-engine, rust

Tags

More Database packages