Grafana Dashboards

Prometheus has its own UI for ad-hoc queries. Grafana is for dashboards — the persistent views your team checks daily, the screens on the wall during an incident.

Data Source Configuration

Provisioned at startup so you never click through setup:

# grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: "15s"          # matches your scrape_interval

For long-term storage (Thanos, Cortex, Mimir), add another data source pointing at it.

Anatomy of a Dashboard

A Grafana dashboard is YAML/JSON containing panels, each with one or more queries and a visualization. The relevant pieces:

• Variables — drop-downs at the top (environment, service, pod) that get templated into every query.
• Time range — global; every panel respects it.
• Refresh — how often panels re-query (5s for incident dashboards, 1m for overview).
• Panels — graphs, stats, gauges, tables, heatmaps.

Panel Types Cheat Sheet

A "Golden Signals" Dashboard

Build one of these for every service. PromQL for each panel:

Variables (the Dropdowns)

Make a dashboard reusable across services and environments by templating:

# Variable: environment
Query type: Label values
Label: environment

# Variable: service (depends on environment)
Query type: Label values
Metric: up{environment="$environment"}
Label: service

Use them in queries:

sum(rate(http_requests_total{environment="$environment", service="$service"}[5m]))

Now the same dashboard works for staging-api, production-api, staging-worker, etc. Pick from the dropdown.

Dashboard Design Principles

A dashboard is a product. A few principles:

1. Top-to-bottom = high level to detail. Service health up top; per-pod drilldown below.
2. Left-to-right = traffic to errors to latency. Roughly the Golden Signals order.
3. One signal per panel. Don't pile six lines onto one graph if they have different units.
4. Y-axis units matter. "Bytes" not "Bytes (1000s)." Grafana auto-formats; help it.
5. Thresholds on stat/gauge panels. Green/yellow/red so anyone can read it at a glance.
6. Annotate releases. Configure a Grafana annotation source so dashboards show deploy markers.
7. Don't overload. A dashboard with 40 panels is two dashboards. Split.

Three Dashboards Every Service Should Have

Service Overview holds Golden Signals plus current alerts and SLO burn. Operational Detail goes deeper — per-endpoint latency, queue depth, downstream dependency health. Infrastructure shows pod/node resource pressure.

Annotations

Grafana annotations are markers drawn across panels — releases, incidents, maintenance windows. Wire them up from your deploy pipeline:

# Add an annotation from CI after a successful deploy
curl -X POST "$GRAFANA_URL/api/annotations" \
  -H "Authorization: Bearer $GRAFANA_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{
    \"dashboardId\": $DASHBOARD_ID,
    \"tags\": [\"deploy\", \"$SERVICE\", \"$ENVIRONMENT\"],
    \"text\": \"Deploy $VERSION\"
  }"

Now during an incident, the question "did this start when we deployed?" answers itself visually.

Versioning Dashboards

Click-and-save in the UI is fine for ad-hoc dashboards. For ones that matter:

• Export to JSON. Commit it to git.
• Provision on startup. grafana/provisioning/dashboards/*.yml points at JSON files; Grafana loads them.
• Review changes. Dashboard changes go through PR review like code.

# grafana/provisioning/dashboards/dashboards.yml
apiVersion: 1
providers:
  - name: 'default'
    folder: ''
    type: file
    options:
      path: /var/lib/grafana/dashboards

Drop the JSON files into /var/lib/grafana/dashboards (mounted from the repo) and they appear in Grafana.

Useful Community Dashboards

Don't build everything from scratch. Grafana Labs publishes dashboards for common exporters:

Import by ID at + → Import dashboard. Use them as a starting point; customize the queries for your environment.

What's Next

You can build dashboards people actually look at. The last page covers running this stack well at scale — cardinality, recording rules, retention, federation → Best Practices.