---
id: pyspark-data-sources
version: "0.1.11"
license: Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for… (full text in the JSON record)
license_treatment: permissive
maintenance: aging
---

# pyspark-data-sources — Custom Spark data sources for reading and writing data in Apache Spark, using the Python Data Source API

License: permissive · Maintenance: aging · Downloads: 78.8K/mo

## What it is and what it does

pyspark-data-sources is a library that wraps the Apache Spark Python Data Source API to simplify reading from and writing to external data systems. It provides pre-built connectors for common sources—including GitHub pull requests, Google Sheets, Kaggle datasets, stock market data, flight tracking, and synthetic data generation via faker—while also offering a framework for building your own custom data sources. The library depends on faker, mkdocstrings, pyarrow, and requests, and is designed for Spark 4.0 and later.

You use it by registering a data source with your Spark session, then reading or writing data using the standard Spark DataFrame API. For example, you can generate synthetic test data on the fly, fetch live flight information, or pull datasets from public APIs without writing low-level Spark connector code. It's intended both as a ready-to-use connector library and as a learning tool for building your own data sources.

Use it for:

- Generate synthetic test data on demand using faker for unit tests and local development.
- Read public datasets from Kaggle, Hugging Face, or GitHub directly into Spark DataFrames.
- Fetch live market data (stocks, crypto, weather) and ingest it into a Spark pipeline.
- Build a custom data source for an internal API or proprietary data system using the provided framework.
- Stream live flight tracking data or other real-time feeds into Spark Structured Streaming jobs.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Provides custom Apache Spark data sources using the Python Data Source API (Spark 4.0+), enabling you to read from and write to external systems directly as Spark DataFrames.

Yes, if you need to integrate external data sources into Spark 4.0+ pipelines and want to avoid writing custom connectors. The library has low install friction and no known vulnerabilities. However, the aging maintenance status (200 days since last release) means you should verify that the specific data sources you need are actively maintained and functional before committing to production use.

## Install

pip install pyspark-data-sources
uv add pyspark-data-sources
poetry add pyspark-data-sources

## Installing pyspark-data-sources

Before you install: Low install friction with a pure-Python wheel. Maintenance status is aging—last release was 200 days ago—so expect slower response to issues, though no known vulnerabilities are present.

License in practice: Apache License 2.0 is permissive; you can use, modify, and distribute this package freely in commercial and private projects, provided you include the license notice and document any changes.

Quickstart:

pip install pyspark-data-sources

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("demo").getOrCreate()
df = spark.read.format("fake").option("numRows", 5).load()
df.show()

Requires Apache Spark 4.0+ or Databricks Runtime 15.4 LTS+, and Python 3.9–3.12.

Verify before relying:

- Whether all listed data sources (GitHub, Google Sheets, Kaggle, etc.) are fully functional or still in development.
- Performance characteristics when reading large datasets from external APIs.
- Whether optional dependencies (huggingface, kaggle, salesforce, lance, robinhood) are well-maintained and compatible.

## Package facts

- License: Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for… (full text in the JSON record) (permissive)
- Python support: capped_below_current
- Install friction: low
- Maintenance: aging
- Downloads: 78.8K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

spark data source python api, custom spark datasources, read external data spark, spark python datasource, fake data generation spark, spark connector builder, apache spark data integration, spark-connector, data-integration, test-data-generation

[View on SkillFed](https://skillfed.io/packages/pyspark-data-sources) · [View on PyPI](https://pypi.org/project/pyspark-data-sources/)