Stand up Kafka and RabbitMQ side-by-side with Docker Compose and send your first message through each

Getting Started

This page boots both Kafka and RabbitMQ on your laptop, then sends and receives one message through each — enough to feel the difference between a log and a broker before you commit to one.

Bring Both Up

# docker-compose.yml
services:
  # --- Kafka (KRaft, no ZooKeeper) ---
  kafka:
    image: confluentinc/cp-kafka:7.6.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:29093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qg"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
    volumes:
      - kafka-data:/var/lib/kafka/data

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    ports: ["8080:8080"]
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
    depends_on: [kafka]

  # --- RabbitMQ ---
  rabbitmq:
    image: rabbitmq:3-management-alpine
    ports:
      - "5672:5672"            # AMQP
      - "15672:15672"          # management UI
    environment:
      RABBITMQ_DEFAULT_USER: user
      RABBITMQ_DEFAULT_PASS: password
    volumes:
      - rabbitmq-data:/var/lib/rabbitmq

volumes:
  kafka-data:
  rabbitmq-data:

docker compose up -d
docker compose ps

Two UIs to bookmark:

Kafka UI → http://localhost:8080
RabbitMQ Management → http://localhost:15672 (user user, password password)

Kafka End-to-End

Create a Topic

docker compose exec kafka kafka-topics \
  --bootstrap-server kafka:9092 \
  --create --topic orders --partitions 3 --replication-factor 1

Produce a Message

docker compose exec -T kafka kafka-console-producer \
  --bootstrap-server kafka:9092 \
  --topic orders <<< 'order-1: 42 widgets'

Consume It

docker compose exec kafka kafka-console-consumer \
  --bootstrap-server kafka:9092 \
  --topic orders --from-beginning --max-messages 1

Now the key Kafka trick — read it again:

# Same command, message still there. That's the log model.
docker compose exec kafka kafka-console-consumer \
  --bootstrap-server kafka:9092 \
  --topic orders --from-beginning --max-messages 1

The message stayed because Kafka retains it. Consumers track their own position; reset to the beginning whenever you want.

From a Real Program (Node.js)

npm install kafkajs

// kafka-demo.js
const { Kafka } = require('kafkajs');

const kafka = new Kafka({ clientId: 'demo', brokers: ['localhost:9092'] });

async function produce() {
  const producer = kafka.producer();
  await producer.connect();
  await producer.send({
    topic: 'orders',
    messages: [{ key: 'order-2', value: JSON.stringify({ id: 2, items: 99 }) }],
  });
  await producer.disconnect();
}

async function consume() {
  const consumer = kafka.consumer({ groupId: 'demo-group' });
  await consumer.connect();
  await consumer.subscribe({ topic: 'orders', fromBeginning: true });
  await consumer.run({
    eachMessage: async ({ partition, message }) => {
      console.log(`partition=${partition} key=${message.key} value=${message.value}`);
    },
  });
}

(async () => {
  await produce();
  await consume();
})();

node kafka-demo.js

You should see both order-1 and order-2 come through — the historical one and the new one.

RabbitMQ End-to-End

Send a Message (CLI via curl)

RabbitMQ's HTTP management API lets you publish without writing code:

curl -u user:password -H 'Content-Type: application/json' \
  -X POST http://localhost:15672/api/exchanges/%2F/amq.default/publish \
  -d '{
    "properties": { "delivery_mode": 2 },
    "routing_key": "hello-queue",
    "payload": "hello from curl",
    "payload_encoding": "string"
  }'

The %2F is URL-encoded /, RabbitMQ's default vhost. delivery_mode: 2 means persistent.

Consume It (Node.js)

npm install amqplib

// rabbit-demo.js
const amqp = require('amqplib');

async function consume() {
  const conn = await amqp.connect('amqp://user:password@localhost:5672');
  const channel = await conn.createChannel();

  await channel.assertQueue('hello-queue', { durable: true });
  await channel.prefetch(1);

  console.log('Waiting for messages...');
  channel.consume('hello-queue', (msg) => {
    if (!msg) return;
    console.log(`received: ${msg.content.toString()}`);
    channel.ack(msg);
  });
}

async function publish() {
  const conn = await amqp.connect('amqp://user:password@localhost:5672');
  const channel = await conn.createChannel();
  await channel.assertQueue('hello-queue', { durable: true });
  channel.sendToQueue('hello-queue', Buffer.from('hello from node'), { persistent: true });
  await channel.close();
  await conn.close();
}

(async () => {
  await publish();
  await consume();   // exits when you ctrl-c
})();

node rabbit-demo.js

Now do the RabbitMQ trick — start a second consumer in another terminal:

node rabbit-demo.js

Publish a few more messages — they get load-balanced across both consumers, one consumer per message. That's the broker model.

If you'd restarted the Kafka consumer instead, it could read the same messages over again. With RabbitMQ, a message acknowledged by one consumer is gone for everyone.

What Just Happened: Side by Side

Step	Kafka	RabbitMQ
"Put a message in"	Append to a partitioned log	Route through an exchange to a queue
"Read it"	Consumer's offset advances	Message taken off the queue
"Read it again"	Reset offset and replay	Gone — it was acked
"Add a consumer"	New consumer group reads from any offset	Existing queue load-balances across consumers
"Different consumer logic, same data"	Different consumer groups, each at their own pace	Fanout exchange + multiple queues

This is the core difference. Everything else — partitioning, exchange types, retention policies, dead-letter queues — flows from it.

Tear Down

docker compose down -v       # -v removes the volumes (deletes data)

What's Next

Each system has deeper details — partitions and consumer groups for Kafka, exchange types and dead-letter handling for RabbitMQ:

Kafka — partitions, consumer groups, producer/consumer code, operational notes
RabbitMQ — exchanges, queues, bindings, dead-letter queues, operational notes
Kafka vs RabbitMQ — side-by-side comparison and selection guide

Getting Started

On this page