Stream Processing

A stream processor runs continuous computation over data that arrives over time — joining, aggregating, transforming, alerting — rather than running batch jobs over data at rest. The output keeps up with the input: when a new event arrives, the relevant computation updates within seconds, not hours.

This is the layer above Message Queues — Kafka holds the events; the stream processor does something useful with them.

Why Stream Processing

The use cases:

• Real-time analytics — dashboards that update as events flow in.
• Fraud / anomaly detection — pattern match in flight.
• Materialized views — keep a denormalized view fresh without full recomputation.
• Enrichment pipelines — join events with reference data, write to a downstream sink.
• Alerting on event patterns — "5 failed logins in 1 minute from the same IP."
• Operational metrics — derive p95 latency from individual request events.
• Change data capture (CDC) — turn database changes into a stream and react.

The Players

For new projects:

• You want SQL and operational simplicity → Materialize or RisingWave.
• JVM team, already on Kafka → Kafka Streams.
• You need maximum power / scale → Flink (self-host or managed).
• You're on AWS / GCP and want managed → Kinesis Data Analytics / Dataflow.
• Python team doing ML / data eng → Bytewax or Spark Structured Streaming.

Stream Processing vs Adjacent Things

You'll often use several together: events flow through Kafka, get processed by Flink, write to Materialize for live queries, and trigger background jobs for side effects.

Learning Path

Two Mental Models

The space splits into two philosophies:

"Stream as a stream"

Flink, Kafka Streams, Spark Streaming. You write code (or SQL) that defines how each event transforms; the framework manages state, windowing, exactly-once. Powerful and flexible; steep learning curve.

// Kafka Streams: count clicks per user per 5-minute window
builder.stream<String, ClickEvent>("clicks")
  .groupBy { _, click -> click.userId }
  .windowedBy(TimeWindows.of(Duration.ofMinutes(5)))
  .count()
  .toStream()
  .to("clicks-per-user-5m")

"Stream as a database"

Materialize, RisingWave. You declare what you want with SQL; the database maintains it incrementally as data flows in.

CREATE MATERIALIZED VIEW clicks_per_user_5m AS
SELECT user_id, COUNT(*) AS click_count
FROM clicks
WHERE created_at > now() - INTERVAL '5 minutes'
GROUP BY user_id;

-- Query like any table; always fresh
SELECT * FROM clicks_per_user_5m ORDER BY click_count DESC LIMIT 10;

For 2026 in non-trivial workloads: start with the streaming-database model unless you have specific needs (huge state, custom logic, integration with Flink ecosystem). It's dramatically easier to reason about.