pyspark-data-sources
Custom Spark data sources for reading and writing data in Apache Spark, using the Python Data Source API
Decision gist · record as of 2026-08-14
Yes, if you need to integrate external data sources into Spark 4.0+ pipelines and want to avoid writing custom connectors. The library has low install friction and no known vulnerabilities. However, the aging maintenance status (200 days since last release) means you should verify that the specific data sources you need are actively maintained and functional before committing to production use.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Apache Spark 4.0+ or Databricks Runtime 15.4 LTS+, and Python 3.9–3.12.
- Low install friction with a pure-Python wheel.
- Maintenance status is aging—last release was 200 days ago—so expect slower response to issues, though no known vulnerabilities are present.
License · maintenance · safety
permissive license (permissive) — Apache License 2.0 is permissive; you can use, modify, and distribute this package freely in commercial and private projects, provided you include the license notice and document any changes.
last release 2026-01-26 (200 days)
0 known vulnerabilities (OSV.dev, 2026-08-14) · 78,772 downloads/mo, #14,410 on PyPI
Alternatives
Verify before relying
pip install pyspark-data-sources
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("demo").getOrCreate()
df = spark.read.format("fake").option("numRows", 5).load()
df.show()- Whether all listed data sources (GitHub, Google Sheets, Kaggle, etc.) are fully functional or still in development.
- Performance characteristics when reading large datasets from external APIs.
- Whether optional dependencies (huggingface, kaggle, salesforce, lance, robinhood) are well-maintained and compatible.
What it is and what it does
pyspark-data-sources is a library that wraps the Apache Spark Python Data Source API to simplify reading from and writing to external data systems. It provides pre-built connectors for common sources—including GitHub pull requests, Google Sheets, Kaggle datasets, stock market data, flight tracking, and synthetic data generation via faker—while also offering a framework for building your own custom data sources. The library depends on faker, mkdocstrings, pyarrow, and requests, and is designed for Spark 4.0 and later.
You use it by registering a data source with your Spark session, then reading or writing data using the standard Spark DataFrame API. For example, you can generate synthetic test data on the fly, fetch live flight information, or pull datasets from public APIs without writing low-level Spark connector code. It's intended both as a ready-to-use connector library and as a learning tool for building your own data sources.
Use it for
- Generate synthetic test data on demand using faker for unit tests and local development.
- Read public datasets from Kaggle, Hugging Face, or GitHub directly into Spark DataFrames.
- Fetch live market data (stocks, crypto, weather) and ingest it into a Spark pipeline.
- Build a custom data source for an internal API or proprietary data system using the provided framework.
- Stream live flight tracking data or other real-time feeds into Spark Structured Streaming jobs.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you need to integrate external data sources into Spark 4.0+ pipelines and want to avoid writing custom connectors.
The library has low install friction and no known vulnerabilities. However, the aging maintenance status (200 days since last release) means you should verify that the specific data sources you need are actively maintained and functional before committing to production use.
Install
pyspark-data-sources on PyPI
Before you install
Low install friction with a pure-Python wheel. Maintenance status is aging—last release was 200 days ago—so expect slower response to issues, though no known vulnerabilities are present.
Requires Apache Spark 4.0+ or Databricks Runtime 15.4 LTS+, and Python 3.9–3.12.
License in practice
Apache License 2.0 is permissive; you can use, modify, and distribute this package freely in commercial and private projects, provided you include the license notice and document any changes.
Quickstart
pip install pyspark-data-sources
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("demo").getOrCreate()
df = spark.read.format("fake").option("numRows", 5).load()
df.show()Verify before relying
- Whether all listed data sources (GitHub, Google Sheets, Kaggle, etc.) are fully functional or still in development.
- Performance characteristics when reading large datasets from external APIs.
- Whether optional dependencies (huggingface, kaggle, salesforce, lance, robinhood) are well-maintained and compatible.
Package facts
| License | permissive license permissive |
| Python support | Capped below the current Python release <3.13,>=3.9 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 4 packagesfakermkdocstringspyarrowrequests |
| Maintenance | Aging 200 days since the last release |
| First released | |
| Downloads | 78,772 / month, #14,410 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
Evidence: pyspark_data_sources-0.1.11-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “spark data source python api”
- pyspark-data-sourcesProvides custom Apache Spark data sources using the Python Data…
- pyspark-huggingfaceRegisters a Spark data source that reads datasets from Hugging Face…
- koalasKoalas implements the pandas DataFrame API on top of Apache Spark,…
Give your agent the search over MCP, or paste the wish link into any chat.
More Distributed Computing packages
gRPC Python is an HTTP/2-based RPC framework that enables you to define and call remote procedures across network boundaries using protocol buffers for serialization.
Install it if you need RPC communication in a distributed system or are integrating with existing gRPC services.
execnet lets you spawn and communicate with Python interpreters across local processes, remote hosts, and different platforms, using a simple API for task distribution and inter-process messaging.
However, the aging maintenance status (275 days since last release) means you should verify it meets your concurrency and performance needs before committing to a…
Cloudpickle extends Python's standard pickle module to serialize lambda functions, interactively-defined functions and classes, and other constructs that the default pickle cannot handle, making it suitable for cluster computing and remote code execution.
Install it if you need to serialize lambda functions, interactively-defined code, or non-standard Python constructs for cluster computing or distributed execution.
Provides a unified, open()-compatible Python API for streaming large files from remote storage (S3, GCS, Azure, HDFS, SFTP, HTTP) and local filesystems, with transparent compression support.
Install it if you work with large files on cloud storage or remote systems and want to avoid writing boilerplate around multiple SDKs.
Portalocker provides cross-platform file locking with support for exclusive and shared locks, plus Redis-based distributed locks and process-aware PID file locking.
Install it if you need file or process coordination; the optional extras (pywin32, redis) are only required for specific lock types.
Ray is a distributed computing framework that scales Python applications from a single machine to multi-node clusters, providing abstractions for parallel tasks, stateful actors, and shared objects.
See also pyspark-huggingface · pyspark-client · dbldatagen · pyspark · sparkmeasure · hyperleaup · sparkdantic · pyspark-test · databricks-sql-connector · databricks-test