pyspark-data-sources

What it is and what it does

pyspark-data-sources is a library that wraps the Apache Spark Python Data Source API to simplify reading from and writing to external data systems. It provides pre-built connectors for common sources—including GitHub pull requests, Google Sheets, Kaggle datasets, stock market data, flight tracking, and synthetic data generation via faker—while also offering a framework for building your own custom data sources. The library depends on faker, mkdocstrings, pyarrow, and requests, and is designed for Spark 4.0 and later.

You use it by registering a data source with your Spark session, then reading or writing data using the standard Spark DataFrame API. For example, you can generate synthetic test data on the fly, fetch live flight information, or pull datasets from public APIs without writing low-level Spark connector code. It's intended both as a ready-to-use connector library and as a learning tool for building your own data sources.

Use it for:

• Generate synthetic test data on demand using faker for unit tests and local development.
• Read public datasets from Kaggle, Hugging Face, or GitHub directly into Spark DataFrames.
• Fetch live market data (stocks, crypto, weather) and ingest it into a Spark pipeline.
• Build a custom data source for an internal API or proprietary data system using the provided framework.
• Stream live flight tracking data or other real-time feeds into Spark Structured Streaming jobs.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides custom Apache Spark data sources using the Python Data Source API (Spark 4.0+), enabling you to read from and write to external systems directly as Spark DataFrames.

Yes, if you need to integrate external data sources into Spark 4.0+ pipelines and want to avoid writing custom connectors. The library has low install friction and no known vulnerabilities. However, the aging maintenance status (200 days since last release) means you should verify that the specific data sources you need are actively maintained and functional before committing to production use.

Install

pyspark-data-sources on PyPI

pip install pyspark-data-sources

uv add pyspark-data-sources

poetry add pyspark-data-sources

Installing pyspark-data-sources

Before you install

Low install friction with a pure-Python wheel. Maintenance status is aging—last release was 200 days ago—so expect slower response to issues, though no known vulnerabilities are present.

License in practice

Apache License 2.0 is permissive; you can use, modify, and distribute this package freely in commercial and private projects, provided you include the license notice and document any changes.

Quickstart

pip install pyspark-data-sources

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("demo").getOrCreate()
df = spark.read.format("fake").option("numRows", 5).load()
df.show()

Requires Apache Spark 4.0+ or Databricks Runtime 15.4 LTS+, and Python 3.9–3.12.

Verify before relying

• Whether all listed data sources (GitHub, Google Sheets, Kaggle, etc.) are fully functional or still in development.
• Performance characteristics when reading large datasets from external APIs.
• Whether optional dependencies (huggingface, kaggle, salesforce, lance, robinhood) are well-maintained and compatible.

Package facts

Evidence: pyspark_data_sources-0.1.11-py3-none-any.whl

Tags

More Distributed Computing packages