{"categories":[{"label":"Database","url":"https://skillfed.io/packages/category/database/3"}],"enrichment":{"capability":"Converts protobuf messages to and from PySpark DataFrames using UDFs, enabling serialization and deserialization of protobuf data in Spark pipelines.","skillfed_tags":["protobuf-serialization","spark-integration"],"use_cases":["Deserialize protobuf-encoded columns in a Spark DataFrame into structured columns for analysis.","Re-encode expanded DataFrame columns back into protobuf binary format for storage or transmission.","Define custom serialization logic for specific protobuf message types within a Spark pipeline.","Convert protobuf data pipelines to Spark for distributed processing without manual schema mapping."],"what_it_does":"pbspark bridges protobuf and PySpark by providing functions to deserialize protobuf-encoded binary data into Spark StructTypes and re-encode them back. It wraps PySpark UDFs to handle the conversion, with special handling for protobuf's bytes, Timestamp, and int64 types to map them correctly to Spark types. The package offers both column-level operations (from_protobuf, to_protobuf) and DataFrame-level helpers (df_from_protobuf, df_to_protobuf) that can optionally expand struct columns into individual fields.\n\nThe MessageConverter class provides a stateful interface for managing conversions and supports custom serializers for non-standard message types. It uses protobuf's MessageToDict internally but overrides its defaults to preserve type fidelity. The package depends on pyspark and protobuf.","worth_installing":"Yes, if you need protobuf-Spark integration and can tolerate abandoned maintenance. The package is stable with no known vulnerabilities, but expect no updates for future pyspark or protobuf incompatibilities. Suitable for existing projects already committed to this stack; risky for new projects expecting long-term support."},"id":"pbspark","links":{"html":"https://skillfed.io/packages/pbspark","md":"https://skillfed.io/packages/pbspark.md","pypi":"https://pypi.org/project/pbspark/"},"maintenance":{"status":"abandoned"},"meta":{"latest_release":"2023-06-07","license_spdx":null,"license_treatment":"permissive","name":"pbspark","python_support":"supports_current","summary":"Convert between protobuf messages and pyspark dataframes"},"popularity":{"monthly_downloads":235040,"position":9012,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.9.0"}