snakebite-py3

What it is and what it does

Snakebite-py3 is a pure Python HDFS client that communicates with Hadoop NameNodes and DataNodes using protobuf. It provides both a library API and a command-line tool for interacting with HDFS clusters. The package is a Python 3 fork maintained by Internet Archive, originally from Spotify, and is designed to work with modern Hadoop versions (2.2.0 and up). It supports most NameNode operations and can read data from DataNodes, with optional CRC checking during transfer (disabled by default for performance).

The library depends on protobuf for serialization and argparse for CLI argument handling. It has been tested primarily against Cloudera CDH5 and Hortonworks HDP2.0 clusters. The package is useful for Python-based applications that need direct programmatic access to HDFS without relying on shell commands.

Use it for:

• Build Python data pipelines that read or write files directly to HDFS clusters without shelling out to commands.
• Automate HDFS file management tasks (listing, checking, reading) from Python scripts in a Hadoop environment.
• Integrate HDFS access into Python applications running on or near a Hadoop cluster for data processing workflows.
• Replace shell-based HDFS access in lightweight Python tools that need to communicate with NameNodes.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides a pure Python HDFS client library and command-line interface for communicating with Hadoop NameNodes and reading data from DataNodes using protobuf.

Yes, if you need programmatic HDFS access from Python and are running Hadoop 2.2.0 or later (CDH5 or HDP2.0+). The package has low install friction, permissive licensing, and no known vulnerabilities. However, maintenance is aging (last release 542 days ago), so verify that it works with your specific Hadoop version before committing to production use.

Install

snakebite-py3 on PyPI

pip install snakebite-py3

uv add snakebite-py3

poetry add snakebite-py3

Installing snakebite-py3

Before you install

Low install friction with only 2 runtime dependencies (protobuf and argparse). Maintenance status is aging—last release was 542 days ago—though the repository remains active with a recent commit on 2026-01-23. This is a maintained fork by Internet Archive for their CDH5 cluster needs.

License in practice

Licensed under Apache License 2.0 (permissive). You can use, modify, and distribute this package freely in commercial and private projects, with minimal restrictions.

Quickstart

pip install snakebite-py3

from snakebite.client import Client

client = Client('namenode_host', namenode_port)
for file_info in client.ls(['/path/to/file']):
    print(file_info)

Requires a running Hadoop cluster; only supports Hadoop 2.2.0 and up (protocol version 9).

Verify before relying

• Whether CRC checking during transfer can be enabled without unacceptable performance impact for your use case.
• Compatibility with Hadoop versions beyond CDH5 and HDP2.0, as the description does not document tested versions.
• Current Python version support (requires_python is unspecified in the fact sheet).
• Specific port number and connection parameters for the Client API.

Package facts

Evidence: snakebite_py3-3.0.6-py3-none-any.whl

Keywords: hadoop, protobuf, hdfs

Tags

More Utilities packages