pbspark

What it is and what it does

pbspark bridges protobuf and PySpark by providing functions to deserialize protobuf-encoded binary data into Spark StructTypes and re-encode them back. It wraps PySpark UDFs to handle the conversion, with special handling for protobuf's bytes, Timestamp, and int64 types to map them correctly to Spark types. The package offers both column-level operations (from_protobuf, to_protobuf) and DataFrame-level helpers (df_from_protobuf, df_to_protobuf) that can optionally expand struct columns into individual fields.

The MessageConverter class provides a stateful interface for managing conversions and supports custom serializers for non-standard message types. It uses protobuf's MessageToDict internally but overrides its defaults to preserve type fidelity. The package depends on pyspark and protobuf.

Use it for:

• Deserialize protobuf-encoded columns in a Spark DataFrame into structured columns for analysis.
• Re-encode expanded DataFrame columns back into protobuf binary format for storage or transmission.
• Define custom serialization logic for specific protobuf message types within a Spark pipeline.
• Convert protobuf data pipelines to Spark for distributed processing without manual schema mapping.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Converts protobuf messages to and from PySpark DataFrames using UDFs, enabling serialization and deserialization of protobuf data in Spark pipelines.

Yes, if you need protobuf-Spark integration and can tolerate abandoned maintenance. The package is stable with no known vulnerabilities, but expect no updates for future pyspark or protobuf incompatibilities. Suitable for existing projects already committed to this stack; risky for new projects expecting long-term support.

Install

pbspark on PyPI

pip install pbspark

uv add pbspark

poetry add pbspark

Installing pbspark

Before you install

Low install friction with a pure-Python wheel. Maintenance is abandoned—last release was 2023-06-07 with no commits since then—so expect no updates for bugs or dependency conflicts.

License in practice

MIT license is permissive; you can use, modify, and distribute this package freely with minimal restrictions.

Quickstart

from pyspark.sql.session import SparkSession
from pbspark import from_protobuf, to_protobuf

spark = SparkSession.builder.getOrCreate()
df_encoded = spark.createDataFrame([{"value": b"..."}])
df_decoded = df_encoded.select(from_protobuf(df_encoded.value, MessageClass).alias("value"))

Requires protobuf message classes generated by protoc; fully qualified module names must match import paths to avoid PicklingError in distributed contexts.

Verify before relying

• Whether abandoned status and lack of recent commits pose compatibility risks with current pyspark or protobuf versions.
• Performance characteristics when handling large message volumes or deeply nested protobuf structures.
• Specific Python version compatibility beyond the stated 3.7–3.11 range.

Package facts

Evidence: pbspark-0.9.0-py3-none-any.whl

Keywords: spark, protobuf, pyspark

Tags

More Database packages