streaming-data
Master real-time data pipelines and event streaming architectures using modern message brokers and processors. This skill covers Kafka, Pulsar, Redpanda, Flink, and Spark across TypeScript, Python, Go, and Java, with patterns for producer/consumer coordination, delivery guarantees, event sourcing, and change data capture.
Streaming Data helps you build event-driven systems using Kafka, Pulsar, Redpanda, Flink, and Spark with producer/consumer patterns.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2025-12-11
Streaming Data helps you build event-driven systems using Kafka, Pulsar, Redpanda, Flink, and Spark with producer/consumer patterns. Master real-time data pipelines and event streaming architectures using modern message brokers and processors. This skill covers Kafka, Pulsar, Redpanda, Flink, and Spark across TypeScript, Python, Go, and Java, with patterns for producer/consumer coordination, delivery guarantees, event sourcing, and change data capture.
Use it when
- streaming-data teaches producer/consumer coordination as a core pattern for event-driven systems.
- streaming-data provides guidance on choosing between Kafka, Pulsar, Redpanda, and other message brokers.
Verify before relying
Read SKILL.md below before installing (16 files). Open directory: indexed for reading, not audited.
Similar skills
Install
ancoleman/ai-design-components/streaming-data · repository language: Python
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
What is streaming-data and what does it cover?
streaming-data is a comprehensive skill for mastering real-time data pipelines and event streaming architectures. It covers modern message brokers including Kafka, Pulsar, and Redpanda, plus stream processors like Flink and Spark. You'll learn producer/consumer patterns, delivery guarantees, event sourcing, change data capture, and implementation across TypeScript, Python, Go, and Java.
How do kafka producer consumer patterns work in streaming-data?
streaming-data teaches producer/consumer coordination as a core pattern for event-driven systems. Producers publish messages to topics; consumers subscribe and process them. The skill covers delivery guarantees (at-least-once, exactly-once), offset management, consumer lag monitoring, and error handling strategies including dead letter queues to ensure reliable message processing.
What message brokers does streaming-data compare?
streaming-data provides guidance on choosing between Kafka, Pulsar, Redpanda, and other message brokers. It helps you evaluate trade-offs in performance, latency, partitioning, and operational complexity. The skill includes setup, configuration, validation, and performance tuning for each platform so you can select the right broker for your architecture.
Can streaming-data help with real-time analytics and microservices?
Yes. streaming-data focuses on implementing real-time data pipelines for analytics and microservices communication. It covers event-driven architecture patterns, microservices coordination via event streaming, IoT real-time analytics, log aggregation, metrics monitoring, and high-frequency trading systems—all using modern stream processors and brokers.
What advanced patterns does streaming-data include?
streaming-data covers advanced patterns like change data capture (CDC) with Debezium, event sourcing for microservices, exactly-once processing with transactions, and error handling strategies. These patterns enable you to build robust, auditable systems with strong delivery semantics and recovery capabilities.
Is streaming-data suitable for TypeScript and Python developers?
Yes. streaming-data provides examples and patterns across TypeScript, Python, Go, and Java. It includes KafkaJS for Node.js, Confluent Kafka Python libraries, and language-specific implementations so you can apply streaming concepts regardless of your primary language.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
Streaming Data Processing
Build production-ready event streaming systems and real-time data pipelines using modern message brokers and stream processors.
When to Use This Skill
Use this skill when: - Building event-driven architectures and microservices communication - Processing real-time analytics, monitoring, or alerting systems - Implementing data integration pipelines (CDC, ETL/ELT) - Creating log or metrics aggregation systems - Developing IoT platforms or high-frequency trading systems
Core Concepts
Message Brokers vs Stream Processors
Message Brokers (Kafka, Pulsar, Redpanda): - Store and distribute event streams - Provide durability, replay capability, partitioning - Handle producer/consumer coordination
Stream Processors (Flink, Spark, Kafka Streams): - Transform and aggregate streaming data -
(truncated - see the full file via the links below)
File tree — 15 files
skills/streaming-data/SKILL.md
skills/streaming-data/examples/python/basic_consumer.py
skills/streaming-data/examples/typescript/basic-producer.ts
skills/streaming-data/outputs.yaml
skills/streaming-data/references/broker-selection.md
skills/streaming-data/references/cdc-patterns.md
skills/streaming-data/references/delivery-guarantees.md
skills/streaming-data/references/error-handling.md
skills/streaming-data/references/event-sourcing.md
skills/streaming-data/references/exactly-once.md
skills/streaming-data/references/go-patterns.md
skills/streaming-data/references/java-patterns.md
skills/streaming-data/references/performance-tuning.md
skills/streaming-data/references/processor-selection.md
skills/streaming-data/references/python-patterns.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Build event-driven systems with message brokers and stream processors”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
Write production-grade integration tests for Kafka producers and consumers using testcontainers to spin up temporary brokers. Test producer/consumer workflows, message ordering, error handling, and delivery semantics in real Kafka environments.
Realtime Sync architects high-concurrency synchronization between distributed clients and servers with sub-50ms latency, using WebTransport bidirectional streams, CRDTs for conflict-free collaborative editing, and transactional outbox patterns for database consistency. Covers presence tracking, sequence-based message recovery, AI token stream batching, and backpressure management to prevent UI jitter.
Build live-updating applications with progressive token streaming for LLM responses, bidirectional chat, and multi-user collaboration. This skill covers protocol selection (SSE for one-way streams, WebSocket for two-way communication, WebRTC for peer-to-peer), conflict-free editing with CRDTs, presence awareness, and offline-first synchronization patterns across Python, Rust, Go, and TypeScript.
Master asynchronous system design through message brokers, event sourcing, and distributed transaction patterns. This skill covers Kafka, RabbitMQ, SQS/SNS integration alongside CQRS, saga orchestration, idempotency strategies, and failure handling with dead letter queues—everything needed to build resilient, decoupled microservice architectures.
Implement event-driven architecture in Spring Boot 3.x using domain events, ApplicationEventPublisher, and Kafka for distributed messaging. This skill covers designing immutable domain events, publishing from aggregate roots, handling events transactionally with @TransactionalEventListener, configuring Kafka producers and consumers, and implementing the outbox pattern for guaranteed event delivery.
Learn the core runtime APIs that power Cloudflare Workers development. This skill covers Fetch for making and handling HTTP requests, Streams for processing large payloads efficiently, Crypto for hashing and signing operations, Cache for response optimization, WebSockets for real-time connections, and text encoding utilities. Includes patterns for timeouts, retries, and error prevention.