---
id: pyarrowfs-adlgen2
version: "0.2.5"
license: MIT
license_treatment: permissive
maintenance: dormant
---

# pyarrowfs-adlgen2 — Use pyarrow with Azure Data Lake gen2

License: permissive · Maintenance: dormant · Downloads: 116.0K/mo

## What it is and what it does

pyarrowfs-adlgen2 bridges PyArrow and Azure Data Lake Gen2 by implementing a PyArrow filesystem that lets you read and write Parquet files directly from cloud storage. Instead of downloading data locally first, you pass the filesystem handler to PyArrow's dataset API and work with remote files as if they were local. It wraps the azure-storage-file-datalake SDK, which provides fast directory listing.

The package is small and stable, supporting Python 3.6 through 3.11. You authenticate via azure.identity, configure optional timeouts, and then use standard PyArrow patterns: read operations with a filesystem argument, or write datasets for PyArrow 3 or greater. The package is dormant but intentionally so—it has a minimal API and no planned major features.

Use it for:

- Read multi-file Parquet datasets from Azure Data Lake Gen2 into PyArrow tables without downloading to local disk.
- Stream large Parquet datasets from Azure for analysis or transformation in memory using PyArrow's columnar operations.
- Write partitioned Parquet datasets back to Azure Data Lake Gen2 directly from PyArrow tables.
- Integrate Azure Data Lake Gen2 as a data source in ETL pipelines that already use PyArrow.
- Access a single container or filesystem within an Azure storage account when full account access is not needed or desired.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Provides a PyArrow filesystem interface for reading and writing Parquet datasets directly from Azure Data Lake Gen2 storage without local copying.

Yes, if you need to read or write Parquet on Azure Data Lake Gen2 with PyArrow. Install friction is low, dependencies are stable, and the MIT license is unrestricted. The dormant maintenance status is not a risk—the package is explicitly stable with no major features planned. The main caveat is that dormancy means no active bug fixes or feature development, so evaluate whether your use case aligns with the current feature set.

## Install

pip install pyarrowfs-adlgen2
uv add pyarrowfs-adlgen2
poetry add pyarrowfs-adlgen2

## Installing pyarrowfs-adlgen2

Before you install: Low friction install with two stable runtime dependencies (pyarrow and azure-storage-file-datalake). Maintenance is dormant—last commit was 2024-06-27 and no releases in 778 days—but the package is explicitly described as stable with no major features planned, so dormancy reflects maturity rather than abandonment.

License in practice: MIT license (permissive) imposes no restrictions on use, modification, or distribution in proprietary or open-source projects.

Quickstart:

pip install pyarrowfs-adlgen2

import azure.identity
import pyarrow.fs
import pyarrowfs_adlgen2

handler = pyarrowfs_adlgen2.AccountHandler.from_account_name(
    'YOUR_ACCOUNT_NAME', azure.identity.DefaultAzureCredential())
fs = pyarrow.fs.PyFileSystem(handler)
ds = pyarrow.dataset.dataset('container/dataset.parq', filesystem=fs)
table = ds.to_table()

Requires Azure credentials configured (e.g., via `az login` or environment variables) for azure.identity.DefaultAzureCredential() to work.

Verify before relying:

- Whether the performance advantage over other filesystem implementations holds across different dataset sizes and structures beyond the NYC taxi benchmark.
- Compatibility with the latest versions of pyarrow and azure-storage-file-datalake beyond what classifiers declare.

## Package facts

- License: MIT (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: dormant
- Downloads: 116.0K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

azure data lake gen2 pyarrow, parquet azure datalake filesystem, read parquet from azure, pyarrow azure storage, azure datalake gen2 connector, azure-cloud, parquet, data-lake

[View on SkillFed](https://skillfed.io/packages/pyarrowfs-adlgen2) · [View on PyPI](https://pypi.org/project/pyarrowfs-adlgen2/)