Stand up Elasticsearch, Logstash, Kibana, and Filebeat with Docker Compose, then ship a log line end-to-end

Getting Started

This page walks one log line through the entire stack — from an application's stdout to a Kibana dashboard. By the end you'll have the pieces in Elasticsearch, Logstash, and Kibana wired together and know how to verify each hop.

Stand Up the Stack

# docker-compose.yml
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ports: ["9200:9200"]
    volumes:
      - es-data:/usr/share/elasticsearch/data
    healthcheck:
      test: ["CMD-SHELL", "curl -s http://localhost:9200/_cluster/health | grep -q '\"status\":\"\\(green\\|yellow\\)\"'"]
      interval: 10s
      retries: 12

  logstash:
    image: docker.elastic.co/logstash/logstash:8.13.0
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
    ports:
      - "5044:5044"                          # beats input
      - "5000:5000"                          # raw TCP for ad-hoc tests
    environment:
      - "LS_JAVA_OPTS=-Xms512m -Xmx512m"
    depends_on:
      elasticsearch:
        condition: service_healthy

  kibana:
    image: docker.elastic.co/kibana/kibana:8.13.0
    ports: ["5601:5601"]
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      elasticsearch:
        condition: service_healthy

  filebeat:
    image: docker.elastic.co/beats/filebeat:8.13.0
    user: root
    volumes:
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    command: ["--strict.perms=false"]
    depends_on:
      - logstash

volumes:
  es-data:

This compose file disables security for local learning (xpack.security.enabled=false). Do not run this configuration anywhere reachable from outside your machine. Production requires TLS and authentication — see Best Practices.

Configure Filebeat to Ship Container Logs

# filebeat/filebeat.yml
filebeat.inputs:
  - type: container
    paths:
      - /var/lib/docker/containers/*/*.log
    processors:
      - add_docker_metadata: ~

output.logstash:
  hosts: ["logstash:5044"]

# Useful for debugging — see what Filebeat is collecting
logging.level: info

Filebeat tails every Docker container's log file and forwards each line to Logstash on port 5044.

Configure a Logstash Pipeline

# logstash/pipeline/main.conf
input {
  beats {
    port => 5044
  }
}

filter {
  # Try to parse JSON logs; if that fails, treat as a plain message
  if [message] =~ /^\{/ {
    json {
      source => "message"
      target => "json"
      skip_on_invalid_json => true
    }
    if [json] {
      # Promote known fields out of the json sub-object
      mutate {
        rename => {
          "[json][level]"      => "level"
          "[json][service]"    => "service"
          "[json][request_id]" => "request_id"
          "[json][duration_ms]" => "duration_ms"
        }
      }
    }
  } else {
    grok {
      match => {
        "message" => "%{TIMESTAMP_ISO8601:ts}\s+%{LOGLEVEL:level}\s+%{DATA:service}\s+%{GREEDYDATA:msg}"
      }
      tag_on_failure => ["_grokparsefailure"]
    }
  }

  # Normalize severity case so dashboards can group cleanly
  if [level] {
    mutate { uppercase => [ "level" ] }
  }

  # Drop noisy Docker daemon chatter
  if [container][name] == "buildkit" {
    drop {}
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "logs-%{+yyyy.MM.dd}"
  }

  # While building the pipeline, also print to stdout so you can see what's happening
  stdout { codec => rubydebug }
}

The pipeline:

Beats input on port 5044 receives events from Filebeat.
Filter branch: parses JSON logs cleanly, falls back to grok for plain-text logs.
Output writes a daily index (logs-2026.05.21) and also dumps to stdout so you can docker compose logs logstash while debugging.

Bring It Up and Verify

docker compose up -d
docker compose ps

# Confirm Elasticsearch is healthy
curl -s http://localhost:9200/_cluster/health?pretty

Generate a few log lines from any container:

docker run --rm --name demo alpine sh -c \
  'for i in 1 2 3; do echo "{\"level\":\"info\",\"service\":\"demo\",\"message\":\"hello $i\"}"; sleep 1; done'

Then look upstream, hop by hop:

# 1. Did Logstash see it? (rubydebug output)
docker compose logs --tail 50 logstash

# 2. Did Elasticsearch index it?
curl -s 'http://localhost:9200/logs-*/_search?pretty&size=3&q=service:demo'

# 3. Are there indices at all?
curl -s 'http://localhost:9200/_cat/indices/logs-*?v'

A working _search response looks like:

{
  "hits": {
    "total": { "value": 3 },
    "hits": [
      {
        "_index": "logs-2026.05.21",
        "_source": {
          "@timestamp": "2026-05-21T00:42:11.234Z",
          "level": "INFO",
          "service": "demo",
          "message": "{\"level\":\"info\",\"service\":\"demo\",\"message\":\"hello 1\"}",
          "container": { "name": "demo", "image": { "name": "alpine" } }
        }
      }
    ]
  }
}

View It in Kibana

Open http://localhost:5601.
Stack Management → Index Patterns → Create: pattern logs-*, time field @timestamp.
Discover: select the logs-* view and pick a time range that includes "now."

You'll see the three hello N events. Add filters like service: "demo" or level: "INFO"; add visible columns for service, level, message. That's your live log explorer.

Build a Trivial Dashboard

In Dashboard → Create:

Visualization	Query	Type
Events over time	`*`	Bar chart, x-axis `@timestamp`, count
Top services	`*`	Pie chart, slice by `service.keyword`
Error count	`level:ERROR`	Metric / single number
Top error messages	`level:ERROR`	Table, group by `message.keyword`, count desc

Save it. Re-running your demo container immediately updates the dashboard.

How the Hops Work

container stdout
   │
   ▼  Docker writes a JSON line per log to /var/lib/docker/containers/<id>/<id>-json.log
Filebeat
   │  Tails those files, adds container metadata, ships to Logstash:5044
   ▼
Logstash
   │  Parses (JSON / grok), normalizes fields, drops noise
   ▼
Elasticsearch  (POST /logs-YYYY.MM.DD/_doc)
   │
   ▼
Kibana queries Elasticsearch on the /_search API and renders Discover + dashboards

When something doesn't show up in Kibana, walk this chain in order: container → Filebeat logs → Logstash stdout → Elasticsearch _cat/indices → Kibana index pattern.

What's Next

Now that the pipeline runs end-to-end, dig into the components:

Elasticsearch — index mappings, queries, cluster sizing, ILM
Logstash — input/filter/output plugins, grok patterns, performance tuning
Kibana — Discover, Lens, dashboards, alerting

For production, also see the Best Practices section of the parent index — TLS, authentication, ILM policies, and Kafka buffering aren't optional once you're past a single host.

Getting Started

On this page