---
id: datafusion
version: "54.0.0"
license: unclear
license_treatment: permissive
maintenance: active
---

# datafusion — Build and run queries against data

License: permissive · Maintenance: active · Downloads: 4.5M/mo

## What it is and what it does

DataFusion is a Python library that wraps Apache Arrow's in-memory query engine, allowing you to execute SQL queries and DataFrame operations against structured data files (Parquet, CSV, JSON) without loading everything into memory at once. It includes a query optimizer that rewrites logical plans for efficiency and supports user-defined Python functions (UDFs and UDAFs) embedded in SQL queries. The library is designed as a foundation for building data systems in Python—projects like Dask SQL and distributed engines (Ballista, Ray) use its Python bindings for SQL parsing and query planning.

You work with DataFusion by creating a SessionContext, registering data sources (files or in-memory DataFrames), and then executing queries either as SQL strings or using a DataFrame API. Results can be collected as PyArrow batches, converted to Pandas DataFrames, or serialized in Substrait format. The package depends on cloudpickle, pyarrow, and typing-extensions, and is actively maintained with prebuilt wheels for common platforms.

Use it for:

- Query large Parquet files with SQL without loading the entire dataset into memory
- Build a SQL interface over CSV or JSON data sources for exploratory analysis
- Register Python UDFs in SQL queries to apply custom logic during query execution
- Convert query results to Pandas DataFrames for visualization or further analysis
- Serialize and deserialize query plans in Substrait format for cross-system compatibility

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

DataFusion is a Python binding to Apache Arrow's in-memory query engine, enabling SQL and DataFrame-based queries against CSV, Parquet, and JSON data with built-in query optimization.

Yes. DataFusion is actively maintained, has no known vulnerabilities, and offers a practical way to execute optimized SQL queries against structured data files in Python. Install it if you need SQL query capabilities over Parquet/CSV/JSON without building a custom query engine or loading entire datasets into memory. The medium install friction is offset by prebuilt wheels and straightforward setup.

## Install

pip install datafusion
uv add datafusion
poetry add datafusion

## Installing datafusion

Before you install: Medium install friction due to compiled wheels; prebuilt binaries available for macOS (x86_64, arm64), Linux (x86_64, aarch64), and Windows (amd64) across Python 3.10–3.14. Active maintenance with recent releases.

License in practice: Licensed under Apache License 2.0 (permissive). Safe for commercial and proprietary use with standard attribution requirements.

Quickstart:

pip install datafusion

from datafusion import SessionContext

ctx = SessionContext()
ctx.register_parquet('taxi', 'data.parquet')
df = ctx.sql('SELECT * FROM taxi WHERE passenger_count > 1')
results = df.collect()

Requires Python 3.10 or later. Parquet/CSV/JSON files must be accessible at query time.

Verify before relying:

- Performance characteristics and typical query latency compared to alternatives
- Memory overhead and scalability limits for large datasets
- Completeness of SQL dialect support and known limitations

## Package facts

- License: not declared (permissive)
- Python support: supports_current
- Install friction: medium
- Maintenance: active
- Downloads: 4.5M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

sql query engine python, parquet query tool, dataframe sql execution, apache arrow python bindings, in-memory query optimizer, csv parquet json queries, python sql dataframe, query-engine, sql-execution, arrow-bindings

[View on SkillFed](https://skillfed.io/packages/datafusion) · [View on PyPI](https://pypi.org/project/datafusion/)