{"categories":[{"label":"Database","url":"https://skillfed.io/packages/category/database/4"}],"enrichment":{"capability":"Provides a PyArrow filesystem interface for reading and writing Parquet datasets directly from Azure Data Lake Gen2 storage without local copying.","skillfed_tags":["azure-cloud","parquet","data-lake"],"use_cases":["Read multi-file Parquet datasets from Azure Data Lake Gen2 into PyArrow tables without downloading to local disk.","Stream large Parquet datasets from Azure for analysis or transformation in memory using PyArrow's columnar operations.","Write partitioned Parquet datasets back to Azure Data Lake Gen2 directly from PyArrow tables.","Integrate Azure Data Lake Gen2 as a data source in ETL pipelines that already use PyArrow.","Access a single container or filesystem within an Azure storage account when full account access is not needed or desired."],"what_it_does":"pyarrowfs-adlgen2 bridges PyArrow and Azure Data Lake Gen2 by implementing a PyArrow filesystem that lets you read and write Parquet files directly from cloud storage. Instead of downloading data locally first, you pass the filesystem handler to PyArrow's dataset API and work with remote files as if they were local. It wraps the azure-storage-file-datalake SDK, which provides fast directory listing.\n\nThe package is small and stable, supporting Python 3.6 through 3.11. You authenticate via azure.identity, configure optional timeouts, and then use standard PyArrow patterns: read operations with a filesystem argument, or write datasets for PyArrow 3 or greater. The package is dormant but intentionally so\u2014it has a minimal API and no planned major features.","worth_installing":"Yes, if you need to read or write Parquet on Azure Data Lake Gen2 with PyArrow. Install friction is low, dependencies are stable, and the MIT license is unrestricted. The dormant maintenance status is not a risk\u2014the package is explicitly stable with no major features planned. The main caveat is that dormancy means no active bug fixes or feature development, so evaluate whether your use case aligns with the current feature set."},"id":"pyarrowfs-adlgen2","links":{"html":"https://skillfed.io/packages/pyarrowfs-adlgen2","md":"https://skillfed.io/packages/pyarrowfs-adlgen2.md","pypi":"https://pypi.org/project/pyarrowfs-adlgen2/"},"maintenance":{"status":"dormant"},"meta":{"latest_release":"2024-06-27","license_spdx":null,"license_treatment":"permissive","name":"pyarrowfs-adlgen2","python_support":"supports_current","summary":"Use pyarrow with Azure Data Lake gen2"},"popularity":{"monthly_downloads":115982,"position":12225,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.2.5"}