Prometheus · Software Engineering Wiki

Cheatsheet #

Question	Query
Error ratio by service	`sum by (service) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (service) (rate(http_requests_total[5m]))`
Request rate	`sum by (service) (rate(http_requests_total[5m]))`
p99 latency	`histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))`
Mean latency	`rate(x_duration_seconds_sum[5m]) / rate(x_duration_seconds_count[5m])`
Errors in the last hour	`increase(http_requests_total{status=~"5.."}[1h])`
Down targets	`up == 0`
Top CPU pods	`topk(5, sum by (pod) (rate(container_cpu_usage_seconds_total[5m])))`
Restarts today	`increase(kube_pod_container_status_restarts_total[24h]) > 0`
Disk full in 4 days	`predict_linear(node_filesystem_free_bytes[6h], 4243600) < 0`
Series per metric	`topk(10, count by (__name__) ({__name__=~".+"}))`
Memory against limit	`sum by (pod) (container_memory_working_set_bytes) / on (pod) group_left sum by (pod) (kube_pod_container_resource_limits{resource="memory"})`
Scrape duration outliers	`topk(10, scrape_duration_seconds)`

Read the ratio next to the rate. A 100% error ratio over two requests is noise; the same ratio over two thousand is an outage.

Vector types #

An instant vector is one sample per series at a point in time; a range vector is every sample in a window. Functions convert between them, and most PromQL errors are a mismatch between the two.

http_requests_total            # instant vector
http_requests_total[5m]        # range vector
rate(http_requests_total[5m])  # range in, instant out
sum(http_requests_total[5m])   # error: sum takes an instant vector

Only instant vectors can be graphed.

Counters, gauges, histograms #

A counter only increases and resets to zero when the process restarts, so its raw value means nothing. rate handles the resets by treating any decrease as a restart.

rate(http_requests_total[5m])      # per-second average across the window, reset-aware
irate(http_requests_total[5m])     # per-second from the last two samples: responsive, noisy
increase(http_requests_total[1h])  # rate × window, for "how many in the last hour"

The window needs at least four scrape intervals or a single missed scrape leaves a gap: [2m] is the floor at a 30s interval, [5m] is the safe default. irate belongs on dashboards, never in alerts.

Gauges move in both directions and can be read directly.

node_memory_MemAvailable_bytes
avg_over_time(node_memory_MemAvailable_bytes[1h])
delta(node_filesystem_free_bytes[1h])
predict_linear(node_filesystem_free_bytes[6h], 4*24*3600) < 0

A classic histogram is a set of cumulative bucket counters labelled le, plus _sum and _count. histogram_quantile interpolates inside the matching bucket, so accuracy is bounded by the bucket boundaries: if the largest finite bucket is 1s and the true p99 is 4s, the answer is wrong at the source.

histogram_quantile(0.99, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))   # correct
rate(sum by (le) (http_request_duration_seconds_bucket)[5m:])                                     # wrong: rate of an aggregate

Aggregate the rates, keep le, then take the quantile.

Quantiles do not average

There is no way to combine p99 across instances by averaging their p99 values. Sum the buckets and compute the quantile from the total.

Selectors and aggregation #

Regex matchers are anchored at both ends, so status=~"5.." matches 500 and not x500.

http_requests_total{job="api", status="500"}
http_requests_total{status=~"5.."}
http_requests_total{status!~"2..|3.."}
{__name__=~"http_.+", job="api"}

sum by (service, status) (rate(http_requests_total[5m]))    # keep these labels
sum without (instance, pod) (rate(http_requests_total[5m])) # drop these, keep the rest
topk(5, sum by (pod) (rate(container_cpu_usage_seconds_total[5m])))
count by (job) (up == 1)

without survives new labels being added upstream; by silently discards them.

Joins #

on (labels) sets the match key, group_left allows many left-hand series per right-hand series, which is how an _info metric contributes labels to a value.

sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))
  * on (pod) group_left (owner_name)
kube_pod_owner

sum by (pod) (container_memory_working_set_bytes)
  / on (pod) group_left
sum by (pod) (kube_pod_container_resource_limits{resource="memory"})

many-to-many matching not allowed means the on set does not uniquely identify a series on one side; add labels to on or aggregate one side first.

Alerting rules #

for is the difference between an alert and a flap: the expression must be true at every evaluation across that window, and one false evaluation resets it.

groups:
  - name: api
    interval: 30s
    rules:
      - alert: ApiHighErrorRate
        expr: |
          sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
            /
          sum by (service) (rate(http_requests_total[5m]))
            > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.service }} returning {{ $value | humanizePercentage }} errors"
          runbook_url: https://wiki.example.internal/runbooks/api-errors

      - alert: TargetDown
        expr: up == 0
        for: 5m
        labels:
          severity: ticket

Alert on symptoms a user can feel — error ratio, latency, saturation of something finite — not on CPU, which is frequently high and fine.

Recording rules #

Precompute what dashboards evaluate repeatedly or alerts evaluate expensively. The level:metric:operation name records what was aggregated away.

groups:
  - name: api-recording
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

Cardinality #

Series count, not sample rate, is what consumes memory: every unique label combination is a separate series with its own in-memory chunk. One unbounded label value kills the server.

# Drop at scrape time, before it costs anything
metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'go_gc_duration_seconds.*'
    action: drop
  - regex: 'request_id'
    action: labeldrop

Never label with a user ID, request ID, resolved URL path, email address or timestamp. A path label is fine as the route template /users/{id} and catastrophic as /users/91823.

Oneliners #

# Targets that are failing, with the reason
curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | select(.health!="up") | [.labels.job, .scrapeUrl, .lastError] | @tsv'

# Biggest metrics by series count
curl -s localhost:9090/api/v1/status/tsdb | jq -r '.data.seriesCountByMetricName[] | [.value, .name] | @tsv'

# Labels with the most values
curl -s localhost:9090/api/v1/status/tsdb | jq -r '.data.labelValueCountByLabelName[] | [.value, .name] | @tsv'

# How many series a metric has right now
curl -s 'localhost:9090/api/v1/series?match[]=http_requests_total' | jq '.data | length'

# Run a query from the shell
curl -s --data-urlencode 'query=sum by (job) (up)' localhost:9090/api/v1/query | jq -r '.data.result[] | [.metric.job, .value[1]] | @tsv'

# Query at a past instant
curl -s --data-urlencode 'query=up' --data-urlencode "time=$(date -d '1 hour ago' +%s)" localhost:9090/api/v1/query | jq '.data.result | length'

# Which rules are failing to evaluate
curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.health!="ok") | [.name, .lastError] | @tsv'

# Firing alerts, grouped
curl -s localhost:9090/api/v1/alerts | jq -r '.data.alerts[] | select(.state=="firing") | [.labels.alertname, .labels.severity] | @tsv' | sort | uniq -c

# Check rule files before reload
promtool check rules rules/*.yaml && promtool check config prometheus.yml

# Reload without restarting (needs --web.enable-lifecycle)
curl -sX POST localhost:9090/-/reload