pyarrowfs-adlgen2

What it is and what it does

pyarrowfs-adlgen2 bridges PyArrow and Azure Data Lake Gen2 by implementing a PyArrow filesystem that lets you read and write Parquet files directly from cloud storage. Instead of downloading data locally first, you pass the filesystem handler to PyArrow's dataset API and work with remote files as if they were local. It wraps the azure-storage-file-datalake SDK, which provides fast directory listing.

The package is small and stable, supporting Python 3.6 through 3.11. You authenticate via azure.identity, configure optional timeouts, and then use standard PyArrow patterns: read operations with a filesystem argument, or write datasets for PyArrow 3 or greater. The package is dormant but intentionally so—it has a minimal API and no planned major features.

Use it for:

• Read multi-file Parquet datasets from Azure Data Lake Gen2 into PyArrow tables without downloading to local disk.
• Stream large Parquet datasets from Azure for analysis or transformation in memory using PyArrow's columnar operations.
• Write partitioned Parquet datasets back to Azure Data Lake Gen2 directly from PyArrow tables.
• Integrate Azure Data Lake Gen2 as a data source in ETL pipelines that already use PyArrow.
• Access a single container or filesystem within an Azure storage account when full account access is not needed or desired.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides a PyArrow filesystem interface for reading and writing Parquet datasets directly from Azure Data Lake Gen2 storage without local copying.

Yes, if you need to read or write Parquet on Azure Data Lake Gen2 with PyArrow. Install friction is low, dependencies are stable, and the MIT license is unrestricted. The dormant maintenance status is not a risk—the package is explicitly stable with no major features planned. The main caveat is that dormancy means no active bug fixes or feature development, so evaluate whether your use case aligns with the current feature set.

Install

pyarrowfs-adlgen2 on PyPI

pip install pyarrowfs-adlgen2

uv add pyarrowfs-adlgen2

poetry add pyarrowfs-adlgen2

Installing pyarrowfs-adlgen2

Before you install

Low friction install with two stable runtime dependencies (pyarrow and azure-storage-file-datalake). Maintenance is dormant—last commit was 2024-06-27 and no releases in 778 days—but the package is explicitly described as stable with no major features planned, so dormancy reflects maturity rather than abandonment.

License in practice

MIT license (permissive) imposes no restrictions on use, modification, or distribution in proprietary or open-source projects.

Quickstart

pip install pyarrowfs-adlgen2

import azure.identity
import pyarrow.fs
import pyarrowfs_adlgen2

handler = pyarrowfs_adlgen2.AccountHandler.from_account_name(
    'YOUR_ACCOUNT_NAME', azure.identity.DefaultAzureCredential())
fs = pyarrow.fs.PyFileSystem(handler)
ds = pyarrow.dataset.dataset('container/dataset.parq', filesystem=fs)
table = ds.to_table()

Requires Azure credentials configured (e.g., via `az login` or environment variables) for azure.identity.DefaultAzureCredential() to work.

Verify before relying

• Whether the performance advantage over other filesystem implementations holds across different dataset sizes and structures beyond the NYC taxi benchmark.
• Compatibility with the latest versions of pyarrow and azure-storage-file-datalake beyond what classifiers declare.

Package facts

Evidence: pyarrowfs_adlgen2-0.2.5-py3-none-any.whl

Keywords: azure, datalake, filesystem, pyarrow, parquet

Tags

More Database packages