pyspark-huggingface

What it is and what it does

pyspark-huggingface is a Spark data source connector that bridges PySpark and Hugging Face's dataset ecosystem. It allows you to stream datasets from Hugging Face directly into Spark DataFrames and write DataFrames back to Hugging Face as Parquet files. The connector supports selecting specific splits and configs, filtering rows and columns, and leverages Hugging Face's Xet deduplication layer to optimize upload speeds. It works with Spark 4 natively and includes a backport for Spark 3.5, 3.4, and 3.3.

The package depends on datasets, huggingface-hub, and pyarrow to handle the actual data transfer and format conversion. Authentication happens via huggingface-cli login or manual token passing. It is designed for distributed, production-grade workflows where you need to ingest or export large-scale datasets between Spark clusters and Hugging Face's storage infrastructure.

Use it for:

• Load public Hugging Face datasets into Spark for distributed ML preprocessing and analysis.
• Export Spark DataFrames as Parquet to Hugging Face for sharing, versioning, or archival.
• Stream specific dataset splits or configs into Spark without downloading the full dataset.
• Apply row and column filters during read to reduce memory footprint and speed up ingestion.
• Use Hugging Face Storage Buckets as a remote data lake, reading and writing via Spark with deduplication.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Registers a Spark data source that reads datasets from Hugging Face and writes Spark DataFrames back to Hugging Face as Parquet, with support for splits, configs, filtering, and deduplication.

Yes. The package is actively maintained, has low install friction, carries no known vulnerabilities, and solves a real integration gap for teams using both Spark and Hugging Face. Install it if you need to move datasets between Spark and Hugging Face, or if you want to use Hugging Face datasets in a Spark pipeline. The Apache License 2.0 poses no restrictions.

Install

pyspark-huggingface on PyPI

pip install pyspark-huggingface

uv add pyspark-huggingface

poetry add pyspark-huggingface

Installing pyspark-huggingface

Before you install

Low friction install with a pure-wheel distribution. Actively maintained as of 2026-04-14. Requires only three runtime dependencies (datasets, huggingface-hub, pyarrow), all widely used in the ML ecosystem.

License in practice

Licensed under Apache License 2.0, a permissive open-source license. You may use, modify, and distribute this package freely in commercial and private projects, provided you include a copy of the license.

Quickstart

pip install pyspark-huggingface

import pyspark_huggingface
df = spark.read.format("huggingface").load("stanfordnlp/imdb")
df.write.format("huggingface").mode("overwrite").save("username/my_dataset")

Requires Python 3.9+. On Spark 3.x, the import statement is required to enable the backport; Spark 4 auto-registers the data source.

Verify before relying

• Minimum and maximum PySpark versions supported beyond the mentioned 3.3, 3.4, 3.5, and 4.
• Performance characteristics for very large datasets or high-frequency read/write operations.
• Specific authentication failure modes and recovery strategies beyond token-based login.
• Whether datasets>=4.8.4 and huggingface-hub>=1.10.1 are hard requirements or only for bucket support.

Package facts

Evidence: pyspark_huggingface-2.1.0-py3-none-any.whl

Tags

More Distributed Computing packages