streaming-data
Master real-time data pipelines and event streaming architectures using modern message brokers and processors. This skill covers Kafka, Pulsar, Redpanda, Flink, and Spark across TypeScript, Python, Go, and Java, with patterns for producer/consumer coordination, delivery guarantees, event sourcing, and change data capture.
Streaming Data helps you build event-driven systems using Kafka, Pulsar, Redpanda, Flink, and Spark with producer/consumer patterns.
AI-generated summary based on this skill's SKILL.md
Install
ancoleman/ai-design-components/streaming-data · repository language: Python
git clone https://github.com/ancoleman/ai-design-components
cp -r ai-design-components/skills/streaming-data ~/.claude/skills/streaming-datanpx skillfed install ancoleman/ai-design-components/streaming-dataFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
What is streaming-data and what does it cover?
streaming-data is a comprehensive skill for mastering real-time data pipelines and event streaming architectures. It covers modern message brokers including Kafka, Pulsar, and Redpanda, plus stream processors like Flink and Spark. You'll learn producer/consumer patterns, delivery guarantees, event sourcing, change data capture, and implementation across TypeScript, Python, Go, and Java.
How do kafka producer consumer patterns work in streaming-data?
streaming-data teaches producer/consumer coordination as a core pattern for event-driven systems. Producers publish messages to topics; consumers subscribe and process them. The skill covers delivery guarantees (at-least-once, exactly-once), offset management, consumer lag monitoring, and error handling strategies including dead letter queues to ensure reliable message processing.
What message brokers does streaming-data compare?
streaming-data provides guidance on choosing between Kafka, Pulsar, Redpanda, and other message brokers. It helps you evaluate trade-offs in performance, latency, partitioning, and operational complexity. The skill includes setup, configuration, validation, and performance tuning for each platform so you can select the right broker for your architecture.
Can streaming-data help with real-time analytics and microservices?
Yes. streaming-data focuses on implementing real-time data pipelines for analytics and microservices communication. It covers event-driven architecture patterns, microservices coordination via event streaming, IoT real-time analytics, log aggregation, metrics monitoring, and high-frequency trading systems—all using modern stream processors and brokers.
What advanced patterns does streaming-data include?
streaming-data covers advanced patterns like change data capture (CDC) with Debezium, event sourcing for microservices, exactly-once processing with transactions, and error handling strategies. These patterns enable you to build robust, auditable systems with strong delivery semantics and recovery capabilities.
Is streaming-data suitable for TypeScript and Python developers?
Yes. streaming-data provides examples and patterns across TypeScript, Python, Go, and Java. It includes KafkaJS for Node.js, Confluent Kafka Python libraries, and language-specific implementations so you can apply streaming concepts regardless of your primary language.
SKILL.md
rendered from the published skill — quoted content, verbatim
Streaming Data Processing
Build production-ready event streaming systems and real-time data pipelines using modern message brokers and stream processors.
When to Use This Skill
Use this skill when: - Building event-driven architectures and microservices communication - Processing real-time analytics, monitoring, or alerting systems - Implementing data integration pipelines (CDC, ETL/ELT) - Creating log or metrics aggregation systems - Developing IoT platforms or high-frequency trading systems
Core Concepts
Message Brokers vs Stream Processors
Message Brokers (Kafka, Pulsar, Redpanda): - Store and distribute event streams - Provide durability, replay capability, partitioning - Handle producer/consumer coordination
Stream Processors (Flink, Spark, Kafka Streams): - Transform and aggregate streaming data -
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 15 files
skills/streaming-data/SKILL.md
skills/streaming-data/examples/python/basic_consumer.py
skills/streaming-data/examples/typescript/basic-producer.ts
skills/streaming-data/outputs.yaml
skills/streaming-data/references/broker-selection.md
skills/streaming-data/references/cdc-patterns.md
skills/streaming-data/references/delivery-guarantees.md
skills/streaming-data/references/error-handling.md
skills/streaming-data/references/event-sourcing.md
skills/streaming-data/references/exactly-once.md
skills/streaming-data/references/go-patterns.md
skills/streaming-data/references/java-patterns.md
skills/streaming-data/references/performance-tuning.md
skills/streaming-data/references/processor-selection.md
skills/streaming-data/references/python-patterns.md