Message Queues
Asynchronous messaging systems for decoupled, scalable architectures - from your first producer to picking the right broker
Message Queues
Message queues let producers and consumers communicate asynchronously. The producer hands a message to the broker and moves on; the consumer reads it whenever it's ready. This decoupling is the foundation of event-driven systems, task pipelines, and most microservice architectures.
Why Use One
| Without a queue | With a queue |
|---|---|
| Service A calls B directly; if B is slow, A blocks | A enqueues, B processes at its own pace |
| If B is down, A's request fails | The message waits in the queue |
| Adding a third consumer requires A to know about it | Multiple consumers subscribe; A doesn't change |
| Bursts overwhelm B | The queue absorbs the burst |
| Retries / dead-letter handling reinvented per call site | The broker provides it once |
Two Mental Models
There are dozens of message systems, but they cluster into two camps:
The Distributed Log (Kafka)
Producer ──► Topic (partition 0) [m1, m2, m3, m4, m5, m6, ...] ──► Consumer Group A
├─► Consumer Group B
──► Topic (partition 1) [m1, m2, m3, m4, m5, m6, ...] ──► Consumer Group A
└─► Consumer Group BMessages are appended to a partitioned log and retained. Consumers track their own position. Multiple consumer groups can read the same messages independently. Replay is built in.
The Message Broker (RabbitMQ)
Producer ──► Exchange ──┬── binding ──► Queue 1 ──► Consumer A
├── binding ──► Queue 2 ──► Consumer B
└── binding ──► Queue 3 ──► Consumer CMessages are routed from exchanges into queues; a consumer takes a message off a queue and acknowledges it; the message is then gone. Flexible routing, but no replay.
The Players
| System | Model | Strength | When to consider |
|---|---|---|---|
| Apache Kafka | Distributed log | Throughput, retention, replay | Event streaming, log pipelines, ≥100K msg/s |
| RabbitMQ | AMQP broker | Routing flexibility, low latency | Task queues, RPC, complex routing |
| Redis Streams | In-memory log | Speed, simplicity | Lightweight messaging when you already run Redis |
| AWS SQS | Managed queue | Zero ops | Simple decoupled cloud workloads |
| NATS | Cloud-native pub/sub | Lightweight, very fast | Microservice signalling, request/reply |
| Pulsar | Hybrid (log + queue) | Tiered storage, multi-tenancy | Want both Kafka and RabbitMQ semantics |
Learning Path
Read in this order if you're new — each page builds on the previous one.
1. Getting Started
Stand up Kafka and RabbitMQ with Docker Compose; send and receive one message end-to-end
2. Kafka
Partitions, consumer groups, producer/consumer code, topic config, operational notes
3. RabbitMQ
Exchanges, queues, bindings, AMQP, dead-letter queues, operational notes
4. Kafka vs RabbitMQ
Side-by-side comparison and selection guide
Concerns That Apply to Every Broker
No matter which system you pick, the same questions come up:
| Concern | What to think about |
|---|---|
| Delivery semantics | At-most-once, at-least-once, exactly-once — pick consciously |
| Ordering | Most systems guarantee ordering only within a partition / queue |
| Idempotent consumers | Networks fail; you'll see the same message twice — design for it |
| Backpressure | What happens when consumers fall behind? Slow producers? Drop? Spool to disk? |
| Dead-letter handling | Where do "poison" messages that always fail go? |
| Schemas | Avro / Protobuf / JSON-Schema — agree before producers and consumers diverge |
| Observability | Lag, throughput, error rate — instrument every producer and consumer |
Message queues sit between services that would otherwise call each other directly. If you need real-time request/response (synchronous, blocking, with a single answer), a queue isn't the right tool — use an HTTP/gRPC call. Queues shine when "I'll get to it eventually" is acceptable.