{"categories":[{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing"}],"enrichment":{"capability":"Python client for connecting to Apache Spark clusters via Spark Connect, enabling distributed data processing and analytics from Python without requiring local Spark JARs.","skillfed_tags":["distributed-computing","data-analytics","spark-connect"],"use_cases":["Run SQL queries and DataFrame transformations against a remote Spark cluster from a Python notebook or script.","Process large datasets distributed across a cluster without installing full Spark locally.","Build machine learning pipelines using MLlib on data too large for a single machine.","Stream and process real-time data using Structured Streaming on a shared Spark cluster.","Integrate Spark analytics into Python applications that connect to an existing enterprise cluster."],"what_it_does":"pyspark-client is the Python Spark Connect client\u2014a remote interface to Apache Spark clusters that lets you submit distributed data processing jobs from Python without needing a local Spark installation. It connects to an existing Spark cluster (standalone, YARN, or Kubernetes) and provides access to Spark SQL, DataFrames, MLlib, and Structured Streaming through a Python API.\n\nThe package depends on pandas, pyarrow, grpcio, grpcio-status, googleapis-common-protos, zstandard, numpy, and pyyaml to handle data serialization, gRPC communication, and numerical computation. It is suitable for interacting with production clusters but does not include tools to set up a standalone cluster itself. The documentation notes that the Python packaging is experimental and may change, and that client and cluster versions must match to avoid runtime errors.","worth_installing":"Yes, if you have access to a running Spark cluster and need Python-based analytics at scale. The high install friction and experimental packaging status are offset by active maintenance, permissive licensing, and broad Python version support (3.10 through 3.14). Critical caveat: ensure your client version matches your cluster version exactly, and verify that all required features are supported via Spark Connect before committing to production use."},"id":"pyspark-client","links":{"html":"https://skillfed.io/packages/pyspark-client","md":"https://skillfed.io/packages/pyspark-client.md","pypi":"https://pypi.org/project/pyspark-client/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-07-14","license_spdx":null,"license_treatment":"permissive","name":"pyspark-client","python_support":"supports_current","summary":"Python Spark Connect client for Apache Spark"},"popularity":{"monthly_downloads":2348591,"position":3115,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"4.2.0"}