{"categories":[{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing/3"}],"enrichment":{"capability":"Provides custom Apache Spark data sources using the Python Data Source API (Spark 4.0+), enabling you to read from and write to external systems directly as Spark DataFrames.","skillfed_tags":["spark-connector","data-integration","test-data-generation"],"use_cases":["Generate synthetic test data on demand using faker for unit tests and local development.","Read public datasets from Kaggle, Hugging Face, or GitHub directly into Spark DataFrames.","Fetch live market data (stocks, crypto, weather) and ingest it into a Spark pipeline.","Build a custom data source for an internal API or proprietary data system using the provided framework.","Stream live flight tracking data or other real-time feeds into Spark Structured Streaming jobs."],"what_it_does":"pyspark-data-sources is a library that wraps the Apache Spark Python Data Source API to simplify reading from and writing to external data systems. It provides pre-built connectors for common sources\u2014including GitHub pull requests, Google Sheets, Kaggle datasets, stock market data, flight tracking, and synthetic data generation via faker\u2014while also offering a framework for building your own custom data sources. The library depends on faker, mkdocstrings, pyarrow, and requests, and is designed for Spark 4.0 and later.\n\nYou use it by registering a data source with your Spark session, then reading or writing data using the standard Spark DataFrame API. For example, you can generate synthetic test data on the fly, fetch live flight information, or pull datasets from public APIs without writing low-level Spark connector code. It's intended both as a ready-to-use connector library and as a learning tool for building your own data sources.","worth_installing":"Yes, if you need to integrate external data sources into Spark 4.0+ pipelines and want to avoid writing custom connectors. The library has low install friction and no known vulnerabilities. However, the aging maintenance status (200 days since last release) means you should verify that the specific data sources you need are actively maintained and functional before committing to production use."},"id":"pyspark-data-sources","links":{"html":"https://skillfed.io/packages/pyspark-data-sources","md":"https://skillfed.io/packages/pyspark-data-sources.md","pypi":"https://pypi.org/project/pyspark-data-sources/"},"maintenance":{"status":"aging"},"meta":{"latest_release":"2026-01-26","license_spdx":null,"license_treatment":"permissive","name":"pyspark-data-sources","python_support":"capped_below_current","summary":"Custom Spark data sources for reading and writing data in Apache Spark, using the Python Data Source API"},"popularity":{"monthly_downloads":78772,"position":14410,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.1.11"}