{"categories":[{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing"}],"enrichment":{"capability":"Registers a Spark data source that reads datasets from Hugging Face and writes Spark DataFrames back to Hugging Face as Parquet, with support for splits, configs, filtering, and deduplication.","skillfed_tags":["spark-integration","hugging-face-datasets","data-pipeline"],"use_cases":["Load public Hugging Face datasets into Spark for distributed ML preprocessing and analysis.","Export Spark DataFrames as Parquet to Hugging Face for sharing, versioning, or archival.","Stream specific dataset splits or configs into Spark without downloading the full dataset.","Apply row and column filters during read to reduce memory footprint and speed up ingestion.","Use Hugging Face Storage Buckets as a remote data lake, reading and writing via Spark with deduplication."],"what_it_does":"pyspark-huggingface is a Spark data source connector that bridges PySpark and Hugging Face's dataset ecosystem. It allows you to stream datasets from Hugging Face directly into Spark DataFrames and write DataFrames back to Hugging Face as Parquet files. The connector supports selecting specific splits and configs, filtering rows and columns, and leverages Hugging Face's Xet deduplication layer to optimize upload speeds. It works with Spark 4 natively and includes a backport for Spark 3.5, 3.4, and 3.3.\n\nThe package depends on datasets, huggingface-hub, and pyarrow to handle the actual data transfer and format conversion. Authentication happens via huggingface-cli login or manual token passing. It is designed for distributed, production-grade workflows where you need to ingest or export large-scale datasets between Spark clusters and Hugging Face's storage infrastructure.","worth_installing":"Yes. The package is actively maintained, has low install friction, carries no known vulnerabilities, and solves a real integration gap for teams using both Spark and Hugging Face. Install it if you need to move datasets between Spark and Hugging Face, or if you want to use Hugging Face datasets in a Spark pipeline. The Apache License 2.0 poses no restrictions."},"id":"pyspark-huggingface","links":{"html":"https://skillfed.io/packages/pyspark-huggingface","md":"https://skillfed.io/packages/pyspark-huggingface.md","pypi":"https://pypi.org/project/pyspark-huggingface/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-04-14","license_spdx":null,"license_treatment":"permissive","name":"pyspark-huggingface","python_support":"supports_current","summary":"A DataSource for reading and writing HuggingFace Datasets in Spark"},"popularity":{"monthly_downloads":900329,"position":4775,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"2.1.0"}