The Prometheus query language - selectors, rate, aggregation, joins, and the patterns you'll use every day

PromQL

PromQL is small but unfamiliar. The mental model: every query returns either an instant vector (one value per series, right now) or a range vector (multiple values per series, over a window).

Selectors and Filtering

# Instant vector — current value of every series
http_requests_total

# Filter by label
http_requests_total{method="GET", status="200"}

# Regex match
http_requests_total{status=~"5.."}

# Negative match
http_requests_total{path!="/health"}

# Multiple matchers (implicit AND)
http_requests_total{method="GET", status=~"2..|3.."}

# Range vector — values over the last 5 minutes
http_requests_total[5m]

Range vectors can't be graphed directly — you have to reduce them to instant vectors with a function like rate().

rate, irate, increase

For counters (monotonically increasing), the raw value isn't interesting — the rate of change is. Three functions:

# Per-second rate, averaged over the last 5 minutes
rate(http_requests_total[5m])

# Per-second rate, using the last two samples only (more reactive, noisier)
irate(http_requests_total[5m])

# Total increase over the window (rate × seconds)
increase(http_requests_total[1h])

When to use which:

rate() — almost always. Use it for graphs, alerts, dashboards.
irate() — only when you need to react quickly to spikes (rare).
increase() — when you want "how many in the last hour" rather than "per second."

The window in rate(x[5m]) must be at least 4× your scrape interval — otherwise you can miss samples and get spiky output. With scrape_interval: 15s, use [1m] or larger.

Aggregation

sum, avg, min, max, count, topk, bottomk, quantile over instant vectors:

# Total request rate across all instances
sum(rate(http_requests_total[5m]))

# Rate broken down by service
sum by (service) (rate(http_requests_total[5m]))

# Rate, dropping these labels
sum without (instance, pod) (rate(http_requests_total[5m]))

# Top 5 pods by memory
topk(5, container_memory_usage_bytes{namespace="production"})

by and without are how you control the grouping — by keeps listed labels, without drops them.

Percentiles from Histograms

Why histograms exist: aggregable percentiles. The trick is always the same shape:

histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

Order matters:

Take the rate of each _bucket series.
sum by (le) collapses other dimensions but keeps le (the bucket boundary).
histogram_quantile interpolates between buckets to estimate the quantile.

Per-service P99:

histogram_quantile(0.99,
  sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))
)

P50, P90, P99 on one chart:

histogram_quantile(0.50, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
histogram_quantile(0.90, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

Common Patterns

Error rate as a percentage

sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
* 100

CPU utilization

100 - (
  avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
  * 100
)

node_cpu_seconds_total is a counter of seconds spent in each mode; the rate is the fraction of a CPU second per real second. idle flipped gives utilization.

Memory utilization

(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)
/
node_memory_MemTotal_bytes
* 100

Disk utilization at /

(node_filesystem_size_bytes{mountpoint="/"} - node_filesystem_avail_bytes{mountpoint="/"})
/
node_filesystem_size_bytes{mountpoint="/"}
* 100

Predict when a disk will fill

predict_linear extrapolates a linear trend forward:

# Will this disk be full in 4 hours? (returns seconds-until-empty; <0 means already full)
predict_linear(node_filesystem_avail_bytes[1h], 4 * 3600) < 0

Great for alerting on slow leaks.

Apdex-style score

Fraction of requests within target latency:

(
  sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
  +
  sum(rate(http_request_duration_seconds_bucket{le="1"}[5m]))
)
/ 2 / sum(rate(http_request_duration_seconds_count[5m]))

Joining Series

PromQL can match labels across two vectors:

# Annotate every up==0 with the instance's friendly name
up{job="api"} == 0
* on(instance) group_left(name)
node_uname_info

on(...) lists labels that must match; group_left(...) brings in extra labels from the right side. This is the trick to enrich alerts with metadata.

Subqueries

A subquery turns an instant query into a range vector — useful when you need a rate-of-a-rate or want to apply aggregation over a longer window:

# Average request rate over the last hour, sampled every minute
avg_over_time(
  sum(rate(http_requests_total[5m]))[1h:1m]
)

The [1h:1m] says "evaluate the inner expression at 1m steps over the last hour."

Reading PromQL — Inside Out

When a query gets long, read it from the innermost selector outward:

sum by (service) (
  rate(
    http_requests_total{status=~"5.."}[5m]
  )
)

Innermost: select the 5xx requests counter.
rate(...[5m]): convert to per-second rate.
sum by (service)(...): total across instances, keep service.

Most "what is this query doing" confusion goes away once you parse it this way.

What's Next

You can ask questions of your metrics. Now write rules that ask them continuously and page you when the answer is wrong → Alerting.

PromQL

On this page