datafusion
Build and run queries against data
Decision gist · record as of 2026-08-14
Yes. DataFusion is actively maintained, has no known vulnerabilities, and offers a practical way to execute optimized SQL queries against structured data files in Python. Install it if you need SQL query capabilities over Parquet/CSV/JSON without building a custom query engine or loading entire datasets into memory. The medium install friction is offset by prebuilt wheels and straightforward setup.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.10 or later.
- Parquet/CSV/JSON files must be accessible at query time.
- Medium install friction due to compiled wheels; prebuilt binaries available for macOS (x86_64, arm64), Linux (x86_64, aarch64), and Windows (amd64) across Python 3.10–3.14.
License · maintenance · safety
permissive license (permissive) — Licensed under Apache License 2.0 (permissive). Safe for commercial and proprietary use with standard attribution requirements.
last release 2026-06-29 (46 days) · last repo commit 2026-08-13 · 595 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 4,474,966 downloads/mo, #2,295 on PyPI
Alternatives
Verify before relying
pip install datafusion
from datafusion import SessionContext
ctx = SessionContext()
ctx.register_parquet('taxi', 'data.parquet')
df = ctx.sql('SELECT * FROM taxi WHERE passenger_count > 1')
results = df.collect()- Performance characteristics and typical query latency compared to alternatives
- Memory overhead and scalability limits for large datasets
- Completeness of SQL dialect support and known limitations
What it is and what it does
DataFusion is a Python library that wraps Apache Arrow's in-memory query engine, allowing you to execute SQL queries and DataFrame operations against structured data files (Parquet, CSV, JSON) without loading everything into memory at once. It includes a query optimizer that rewrites logical plans for efficiency and supports user-defined Python functions (UDFs and UDAFs) embedded in SQL queries. The library is designed as a foundation for building data systems in Python—projects like Dask SQL and distributed engines (Ballista, Ray) use its Python bindings for SQL parsing and query planning.
You work with DataFusion by creating a SessionContext, registering data sources (files or in-memory DataFrames), and then executing queries either as SQL strings or using a DataFrame API. Results can be collected as PyArrow batches, converted to Pandas DataFrames, or serialized in Substrait format. The package depends on cloudpickle, pyarrow, and typing-extensions, and is actively maintained with prebuilt wheels for common platforms.
Use it for
- Query large Parquet files with SQL without loading the entire dataset into memory
- Build a SQL interface over CSV or JSON data sources for exploratory analysis
- Register Python UDFs in SQL queries to apply custom logic during query execution
- Convert query results to Pandas DataFrames for visualization or further analysis
- Serialize and deserialize query plans in Substrait format for cross-system compatibility
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
DataFusion is actively maintained, has no known vulnerabilities, and offers a practical way to execute optimized SQL queries against structured data files in Python. Install it if you need SQL query capabilities over Parquet/CSV/JSON without building a custom query engine or loading entire datasets into memory. The medium install friction is offset by prebuilt wheels and straightforward setup.
Install
datafusion on PyPI
Before you install
Medium install friction due to compiled wheels; prebuilt binaries available for macOS (x86_64, arm64), Linux (x86_64, aarch64), and Windows (amd64) across Python 3.10–3.14. Active maintenance with recent releases.
Requires Python 3.10 or later. Parquet/CSV/JSON files must be accessible at query time.
License in practice
Licensed under Apache License 2.0 (permissive). Safe for commercial and proprietary use with standard attribution requirements.
Quickstart
pip install datafusion
from datafusion import SessionContext
ctx = SessionContext()
ctx.register_parquet('taxi', 'data.parquet')
df = ctx.sql('SELECT * FROM taxi WHERE passenger_count > 1')
results = df.collect()Verify before relying
- Performance characteristics and typical query latency compared to alternatives
- Memory overhead and scalability limits for large datasets
- Completeness of SQL dialect support and known limitations
Package facts
| License | permissive license permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Medium. Platform-specific wheel |
| Runtime dependencies | 3 packagescloudpicklepyarrowtyping-extensions |
| Maintenance | Actively maintained 46 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 4,474,966 / month, #2,295 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 2 - Pre-AlphaIntended Audience :: DevelopersLicense :: OSI ApprovedLicense :: OSI Approved :: Apache Software LicenseOperating System :: MacOSOperating System :: Microsoft :: WindowsOperating System :: POSIX :: LinuxProgramming Language :: PythonProgramming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Programming Language :: Python :: Free Threading :: 2 - BetaProgramming Language :: Rust |
Evidence: datafusion-54.0.0-cp310-abi3-macosx_10_12_x86_64.whl; datafusion-54.0.0-cp310-abi3-macosx_11_0_arm64.whl; datafusion-54.0.0-cp310-abi3-manylinux_2_28_aarch64.whl; datafusion-54.0.0-cp310-abi3-manylinux_2_28_x86_64.whl; datafusion-54.0.0-cp310-abi3-win_amd64.whl; datafusion-54.0.0-cp314-cp314t-macosx_10_12_x86_64.whl; datafusion-54.0.0-cp314-cp314t-macosx_11_0_arm64.whl; datafusion-54.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl; datafusion-54.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl; datafusion-54.0.0-cp314-cp314t-win_amd64.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “parquet query tool”
- datafusionDataFusion is a Python binding to Apache Arrow's in-memory query…
- arcosparseDownloads and subsets sparse geospatial datasets stored in ARCO…
- parquetReads Apache Parquet files in pure Python and outputs data as JSON or…
Give your agent the search over MCP, or paste the wish link into any chat.
More Database packages
psycopg2-binary is a PostgreSQL database adapter for Python that implements the DB API 2.0 specification, enabling Python applications to connect to and query PostgreSQL databases with thread-safe concurrent operations.
Python client library for connecting to and executing commands against Redis key-value stores, supporting both synchronous and asynchronous operations.
Install it if your application needs to interact with Redis; the only prerequisite is a running Redis server instance.
YDB Python SDK is the official client library for connecting to and querying YDB databases from Python applications.
Install it if you need to connect Python applications to YDB databases.
Connects Python applications to Snowflake data warehouses using the DB API 2.0 specification, enabling SQL queries, data transfers, and warehouse operations.
sqlparse tokenizes SQL text into a tree of statements, clauses, and expressions, and provides functions to split scripts, format queries, and inspect parsed tokens without validating dialect or syntax.
Install it if you need to manipulate, format, or analyze SQL text programmatically.
Provides base adapter protocols and shared functionality that database adapters use to integrate with dbt-core, handling connections, dialect translation, relation caching, and core interface management.
See also datafusion-query-builder · pyarrow · vortex-data · ibis-framework · substrait · apache-airflow-providers-common-sql · duckdb · pandasql · polars · fastparquet