{"categories":[{"label":"Database","url":"https://skillfed.io/packages/category/database"}],"enrichment":{"capability":"DataFusion is a Python binding to Apache Arrow's in-memory query engine, enabling SQL and DataFrame-based queries against CSV, Parquet, and JSON data with built-in query optimization.","skillfed_tags":["query-engine","sql-execution","arrow-bindings"],"use_cases":["Query large Parquet files with SQL without loading the entire dataset into memory","Build a SQL interface over CSV or JSON data sources for exploratory analysis","Register Python UDFs in SQL queries to apply custom logic during query execution","Convert query results to Pandas DataFrames for visualization or further analysis","Serialize and deserialize query plans in Substrait format for cross-system compatibility"],"what_it_does":"DataFusion is a Python library that wraps Apache Arrow's in-memory query engine, allowing you to execute SQL queries and DataFrame operations against structured data files (Parquet, CSV, JSON) without loading everything into memory at once. It includes a query optimizer that rewrites logical plans for efficiency and supports user-defined Python functions (UDFs and UDAFs) embedded in SQL queries. The library is designed as a foundation for building data systems in Python\u2014projects like Dask SQL and distributed engines (Ballista, Ray) use its Python bindings for SQL parsing and query planning.\n\nYou work with DataFusion by creating a SessionContext, registering data sources (files or in-memory DataFrames), and then executing queries either as SQL strings or using a DataFrame API. Results can be collected as PyArrow batches, converted to Pandas DataFrames, or serialized in Substrait format. The package depends on cloudpickle, pyarrow, and typing-extensions, and is actively maintained with prebuilt wheels for common platforms.","worth_installing":"Yes. DataFusion is actively maintained, has no known vulnerabilities, and offers a practical way to execute optimized SQL queries against structured data files in Python. Install it if you need SQL query capabilities over Parquet/CSV/JSON without building a custom query engine or loading entire datasets into memory. The medium install friction is offset by prebuilt wheels and straightforward setup."},"id":"datafusion","links":{"html":"https://skillfed.io/packages/datafusion","md":"https://skillfed.io/packages/datafusion.md","pypi":"https://pypi.org/project/datafusion/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-06-29","license_spdx":null,"license_treatment":"permissive","name":"datafusion","python_support":"supports_current","summary":"Build and run queries against data"},"popularity":{"monthly_downloads":4474966,"position":2295,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"54.0.0"}