---
id: pbspark
version: "0.9.0"
license: MIT
license_treatment: permissive
maintenance: abandoned
---

# pbspark — Convert between protobuf messages and pyspark dataframes

License: permissive · Maintenance: abandoned · Downloads: 235.0K/mo

## What it is and what it does

pbspark bridges protobuf and PySpark by providing functions to deserialize protobuf-encoded binary data into Spark StructTypes and re-encode them back. It wraps PySpark UDFs to handle the conversion, with special handling for protobuf's bytes, Timestamp, and int64 types to map them correctly to Spark types. The package offers both column-level operations (from_protobuf, to_protobuf) and DataFrame-level helpers (df_from_protobuf, df_to_protobuf) that can optionally expand struct columns into individual fields.

The MessageConverter class provides a stateful interface for managing conversions and supports custom serializers for non-standard message types. It uses protobuf's MessageToDict internally but overrides its defaults to preserve type fidelity. The package depends on pyspark and protobuf.

Use it for:

- Deserialize protobuf-encoded columns in a Spark DataFrame into structured columns for analysis.
- Re-encode expanded DataFrame columns back into protobuf binary format for storage or transmission.
- Define custom serialization logic for specific protobuf message types within a Spark pipeline.
- Convert protobuf data pipelines to Spark for distributed processing without manual schema mapping.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Converts protobuf messages to and from PySpark DataFrames using UDFs, enabling serialization and deserialization of protobuf data in Spark pipelines.

Yes, if you need protobuf-Spark integration and can tolerate abandoned maintenance. The package is stable with no known vulnerabilities, but expect no updates for future pyspark or protobuf incompatibilities. Suitable for existing projects already committed to this stack; risky for new projects expecting long-term support.

## Install

pip install pbspark
uv add pbspark
poetry add pbspark

## Installing pbspark

Before you install: Low install friction with a pure-Python wheel. Maintenance is abandoned—last release was 2023-06-07 with no commits since then—so expect no updates for bugs or dependency conflicts.

License in practice: MIT license is permissive; you can use, modify, and distribute this package freely with minimal restrictions.

Quickstart:

from pyspark.sql.session import SparkSession
from pbspark import from_protobuf, to_protobuf

spark = SparkSession.builder.getOrCreate()
df_encoded = spark.createDataFrame([{"value": b"..."}])
df_decoded = df_encoded.select(from_protobuf(df_encoded.value, MessageClass).alias("value"))

Requires protobuf message classes generated by protoc; fully qualified module names must match import paths to avoid PicklingError in distributed contexts.

Verify before relying:

- Whether abandoned status and lack of recent commits pose compatibility risks with current pyspark or protobuf versions.
- Performance characteristics when handling large message volumes or deeply nested protobuf structures.
- Specific Python version compatibility beyond the stated 3.7–3.11 range.

## Package facts

- License: MIT (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: abandoned
- Downloads: 235.0K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

protobuf pyspark conversion, protobuf dataframe serialization, spark protobuf udf, protobuf to dataframe, pyspark protobuf integration, protobuf-serialization, spark-integration

[View on SkillFed](https://skillfed.io/packages/pbspark) · [View on PyPI](https://pypi.org/project/pbspark/)