{"categories":[{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing/3"}],"enrichment":{"capability":"Provides Apache Beam transforms for reading and writing relational databases, consuming and producing Kafka messages, and parsing CSV files and JSON data.","skillfed_tags":["data-pipeline","database-io","kafka-integration"],"use_cases":["Read records from a PostgreSQL or MySQL table and process them in a Beam pipeline.","Write transformed data from a Beam pipeline directly into a relational database table.","Consume messages from a Kafka topic and process them in a Beam pipeline.","Produce processed records to a Kafka topic from a Beam pipeline.","Parse CSV files or JSON data as part of a Beam data processing workflow."],"what_it_does":"beam-nuggets is a collection of custom transforms that extend Apache Beam's Python SDK with connectors for relational databases, Kafka, and CSV files. It wraps SQLAlchemy to provide ReadFromDB and Write transforms that work with any SQLAlchemy-supported database (PostgreSQL, MySQL, SQLite tested), plus KafkaConsume and KafkaProduce for Kafka integration and CSV reading. It also includes utility transforms for parsing JSON, selecting from nested dictionaries, and assigning unique IDs.\n\nThe package is most useful for data pipelines that need to ingest from or load into SQL databases or Kafka topics within a Beam pipeline. It abstracts away the boilerplate of configuring database connections and serialization, letting you focus on pipeline logic. However, the project is dormant\u2014last updated in September 2021\u2014so it may not work with recent versions of Beam, database drivers, or Kafka clients without manual fixes.","worth_installing":"Yes, if you need Beam-to-database or Beam-to-Kafka connectors and can tolerate dormant maintenance. The package has low install friction and no known vulnerabilities, but verify compatibility with your Beam and driver versions first. If you require active support or need to integrate with very recent Beam or database driver releases, consider forking or maintaining a local patch."},"id":"beam-nuggets","links":{"html":"https://skillfed.io/packages/beam-nuggets","md":"https://skillfed.io/packages/beam-nuggets.md","pypi":"https://pypi.org/project/beam-nuggets/"},"maintenance":{"status":"dormant"},"meta":{"latest_release":"2021-09-12","license_spdx":null,"license_treatment":"permissive","name":"beam-nuggets","python_support":"unspecified","summary":"Collection of transforms for the Apache beam python SDK."},"popularity":{"monthly_downloads":76789,"position":14590,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.18.1"}