hdfs

What it is and what it does

HdfsCLI is a Python library and CLI tool that wraps the WebHDFS API to let you interact with Hadoop Distributed File System clusters. It provides both programmatic access—listing directories, reading and writing files, checking file metadata—and an interactive shell for ad-hoc exploration. The package supports both secure and insecure clusters, and includes optional extensions for Avro serialization and Pandas dataframe integration.

The library is marked Production/Stable and supports Python 3.7, 3.8, 3.9, 3.10, 3.11, and 3.12, but has not been actively maintained since 2023-10-13. It carries no known security vulnerabilities. Installation requires building from source, which adds friction. If your HDFS cluster is stable and your use case is straightforward file operations, the dormant status may not matter; if you need active support or are integrating with rapidly evolving Hadoop ecosystems, the lack of maintenance is a significant risk.

Use it for:

• Batch upload or download files to/from HDFS in Python scripts without writing Java code.
• Query HDFS file metadata (permissions, replication, block size) programmatically from Python.
• Load Pandas dataframes directly from HDFS files using the optional dataframe extension.
• Automate HDFS file operations in data pipelines using the Python API.
• Interactively explore HDFS directory structure and file contents via the CLI shell.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

HdfsCLI provides Python bindings and a command-line interface for interacting with HDFS clusters via the WebHDFS API, supporting file operations, metadata queries, and an interactive shell.

Yes, if you have a stable HDFS cluster and need straightforward Python access to it. The MIT license is permissive, there are no known vulnerabilities, and the API is mature. However, the package is dormant, so expect no active support or fixes for new Hadoop versions. Suitable for legacy systems or simple use cases; risky for new projects expecting ongoing maintenance.

Install

hdfs on PyPI

pip install hdfs

uv add hdfs

poetry add hdfs

Installing hdfs

Before you install

Installation requires a source tarball (hdfs-2.7.3.tar.gz) with high friction. The package is dormant—last release was 1036 days ago—so expect no active maintenance or bug fixes going forward.

License in practice

MIT license is permissive, allowing commercial and private use with minimal restrictions, making this safe from a licensing standpoint for most projects.

Quickstart

pip install hdfs

from hdfs import InsecureClient
client = InsecureClient('http://localhost:50070')
files = client.list('/path')
with client.read('/path/file.txt') as reader:
    content = reader.read()

Requires an accessible HDFS namenode or HttpFS endpoint; WebHDFS must be enabled on the cluster.

Verify before relying

• Whether optional extensions (avro, dataframe, kerberos) are installable and functional given the dormant maintenance status.
• Current compatibility with modern Hadoop/HDFS versions and whether breaking API changes have occurred since the last release.
• Whether the package remains compatible with Python 3.10, 3.11, and 3.12 despite dormant maintenance status.

Package facts

Evidence: hdfs-2.7.3.tar.gz

Tags

More Distributed Computing packages