Best Practices

Stream processors look declarative but operate stateful, latency-sensitive, hard-to-recover systems. These habits keep them well-behaved.

Bound Your State

The single most important rule. State grows monotonically unless you bound it.

Bound state via:

A pipeline without bounded state will eventually OOM or fall over recovering. Set explicit limits at design time.

Watermark Tuning

Watermarks are the trickiest knob:

• Too aggressive → late events dropped (incorrect results).
• Too conservative → high latency (slow downstream updates).

Start by measuring lateness in production data:

P50 lateness: 100 ms
P95 lateness: 2 s
P99 lateness: 30 s
P99.9 lateness: 5 min

Set the watermark slack at P99 to P99.9 — most events handled correctly, with acceptable lateness budget for stragglers. Configure a side-output for events past the watermark; you can re-process or alert on volume.

Checkpointing and Recovery

Every stateful stream processor must checkpoint to recover from failure:

Test recovery before going to production. Kill a worker; verify the pipeline catches up correctly.

Latency Budget

Define latency targets explicitly:

The latency budget shapes:

• Watermark slack (faster = less slack for late events).
• Checkpoint interval (longer = more recovery latency).
• Source poll interval (Kafka consumer wait).
• Sink write strategy (batch vs streaming).

Monitor end-to-end latency, not just per-stage. Event time at source → event observed in sink is what users see.

Observability

What to monitor:

Flink ships Prometheus metrics. Materialize has built-in observability views. Kafka Streams exposes JMX metrics. Pipe everything to Prometheus & Grafana.

Schema Evolution

Streams live forever; schemas don't:

• Use a schema registry (Confluent Schema Registry, Apicurio).
• Avro / Protobuf / JSON Schema with documented evolution rules.
• Backward compatibility: new schema can read old events.
• Forward compatibility: old schema can read new events.
• Both = "full compatibility" — strongest, what you want for long-lived pipelines.

A breaking schema change in production stops your pipeline. Test schema changes against historical data before deploy.

Deployment

Stream pipelines are infrastructure, not just applications:

Some pipelines can replay from Kafka's beginning to rebuild state — others have too much. Know which yours is before you need to recover.

Scale Considerations

For state size: in-memory works up to a few GB; RocksDB on local disk handles hundreds of GB to TBs. Beyond that, partition aggressively.

Cost

Stream processors are continuously running compute. Costs:

Managed services (Confluent Flink, Materialize Cloud, AWS Kinesis Data Analytics) cost more per unit but include operations. The cheapest option is rarely the cheapest total cost of ownership.

When NOT to Use Stream Processing

Honest cases:

• You can run a batch job hourly and that's fine for the use case. Don't reach for streaming if you don't need the latency.
• You have one consumer that needs to react to events — a Background Job consumer with idempotency is simpler.
• You have a few aggregations that can live in your OLTP DB — Postgres can maintain a counter via triggers.
• You don't have someone to own the pipeline. Stream processors are stateful systems that need care; underestimating that has burned every team that tried.

If you have 1-2 high-value pipelines (real-time analytics, materialized views, fraud), stream processing is worth it. If you're rebuilding your entire application around streaming because it's the new shiny, stop.

Common Pitfalls

Checklist