Software Engineering Wiki

Data

Elasticsearch

Cluster and index health, query and aggregation syntax, mappings, and the checks for unassigned shards or a slow search.

Cheatsheet #

TaskRequest
Cluster healthGET _cluster/health?pretty
Why is it yellow or redGET _cluster/allocation/explain
Indices by sizeGET _cat/indices?v&s=store.size:desc
Shard placementGET _cat/shards?v&s=state
Node resourcesGET _cat/nodes?v&h=name,heap.percent,cpu,load_1m,disk.used_percent
Pending tasksGET _cat/pending_tasks?v
Long-running tasksGET _tasks?actions=*search&detailed
Mapping of an indexGET myindex/_mapping
Count matchingGET myindex/_count with a query
Explain a query’s scoreGET myindex/_explain/<id>
Validate a queryGET myindex/_validate/query?explain
Analyse textGET myindex/_analyze
Force a refreshPOST myindex/_refresh
Cluster settingsGET _cluster/settings?include_defaults&flat_settings

Health and shards #

Green means every primary and replica is assigned; yellow means replicas are missing; red means a primary is missing and part of the data is unreadable. The allocation explain API states the exact reason rather than leaving it to inference.

curl -s localhost:9200/_cluster/health?pretty
curl -s 'localhost:9200/_cat/indices?v&health=red&s=index'
curl -s localhost:9200/_cluster/allocation/explain?pretty -H 'content-type: application/json' -d '{
  "index": "logs-2024.09.01", "shard": 0, "primary": true
}'
curl -s 'localhost:9200/_cat/shards?v&s=state&h=index,shard,prirep,state,node,unassigned.reason'
Unassigned reasonMeaning
INDEX_CREATEDNormal, briefly, after creation
NODE_LEFTA node disappeared; recovery in progress or blocked
ALLOCATION_FAILEDRetries exhausted — POST _cluster/reroute?retry_failed=true
NO_VALID_SHARD_COPYData is gone; only a restore or forced allocation recovers it
DISK_THRESHOLDWatermark exceeded; free space or adjust the watermark

Disk watermarks (85% low, 90% high, 95% flood stage) move or freeze shards. The flood stage sets indices read-only, which presents as writes failing while the cluster looks green.

curl -s -XPUT localhost:9200/_cluster/settings -H 'content-type: application/json' -d '{
  "persistent": {"cluster.routing.allocation.disk.watermark.flood_stage": "97%"}
}'
curl -s -XPUT 'localhost:9200/myindex/_settings' -H 'content-type: application/json' -d '{"index.blocks.read_only_allow_delete": null}'

Indexing and mapping #

A mapping decides how text is analysed and therefore what can be searched. text is analysed into terms for full-text matching; keyword is stored whole for exact matching, sorting and aggregations. Most “why does my term query return nothing” questions are this distinction.

PUT myindex
{
  "settings": { "number_of_shards": 1, "number_of_replicas": 1, "refresh_interval": "5s" },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "service":   { "type": "keyword" },
      "message":   { "type": "text", "analyzer": "standard" },
      "level":     { "type": "keyword" },
      "duration_ms": { "type": "float" },
      "@timestamp": { "type": "date" },
      "host": {
        "properties": { "name": { "type": "keyword" }, "ip": { "type": "ip" } }
      }
    }
  }
}

Mappings are immutable for existing fields: changing a type requires a new index and _reindex. dynamic: strict rejects unexpected fields instead of guessing a type and locking it in.

curl -XPOST localhost:9200/myindex/_doc -H 'content-type: application/json' -d '{"service":"api","message":"timeout"}'
curl -XPUT  localhost:9200/myindex/_doc/1 -H 'content-type: application/json' -d '{"service":"api"}'
curl -XPOST localhost:9200/myindex/_update/1 -H 'content-type: application/json' -d '{"doc":{"level":"error"}}'
curl -XPOST localhost:9200/_bulk -H 'content-type: application/x-ndjson' --data-binary @bulk.ndjson
{"index":{"_index":"myindex","_id":"1"}}
{"service":"api","message":"timeout","@timestamp":"2024-09-01T00:00:00Z"}

Bulk is the only sane way to index volume: one request per document wastes a round trip and a refresh cycle each time. Newlines matter, including the trailing one.

Searching #

GET myindex/_search
{
  "size": 20,
  "track_total_hits": true,
  "query": {
    "bool": {
      "must":   [{ "match": { "message": "connection timeout" } }],
      "filter": [
        { "term":  { "service": "api" } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ],
      "must_not": [{ "term": { "level": "debug" } }],
      "should":  [{ "match_phrase": { "message": "connection refused" } }]
    }
  },
  "sort": [{ "@timestamp": "desc" }],
  "_source": ["service", "message", "@timestamp"]
}
ClauseEffect
mustMust match, contributes to the score
filterMust match, no score, cacheable — use it for everything structured
shouldBoosts when matched
must_notExcludes, no score
matchAnalysed full-text search
termExact, unanalysed — on a text field it will usually match nothing
match_phraseTerms in order
wildcard / regexpExpensive; avoid leading wildcards entirely

Put date ranges and identifiers in filter, not must: filters skip scoring and are cached per segment, which is often an order of magnitude cheaper.

GET myindex/_search
{
  "size": 0,
  "aggs": {
    "by_service": {
      "terms": { "field": "service", "size": 10 },
      "aggs": {
        "p95": { "percentiles": { "field": "duration_ms", "percents": [95] } },
        "over_time": { "date_histogram": { "field": "@timestamp", "fixed_interval": "5m" } }
      }
    }
  }
}

size: 0 skips returning documents when only aggregations are wanted. terms aggregations on high-cardinality fields are memory-hungry — use composite for pagination over many buckets.

Reindex, aliases and lifecycle #

Aliases let an index be replaced without changing the client. Write to an alias, reindex into a new index, then swap atomically.

curl -XPOST localhost:9200/_reindex -H 'content-type: application/json' -d '{
  "source": {"index": "myindex"}, "dest": {"index": "myindex-v2"}
}'
curl -XPOST localhost:9200/_aliases -H 'content-type: application/json' -d '{
  "actions": [
    {"remove": {"index": "myindex", "alias": "myindex-current"}},
    {"add":    {"index": "myindex-v2", "alias": "myindex-current"}}
  ]
}'

For time-series data use data streams with an ILM policy: hot for writes, warm for search, delete at a fixed age. Nothing should be deleting log indices with a cron job and a date pattern.

curl -s localhost:9200/_ilm/policy/logs?pretty
curl -s 'localhost:9200/_cat/indices/logs-*?v&s=index' | tail
curl -s localhost:9200/myindex/_ilm/explain?pretty

Performance #

curl -s 'localhost:9200/_nodes/stats/jvm,indices?pretty' | jq '.nodes[] | {heap: .jvm.mem.heap_used_percent, gc: .jvm.gc.collectors.old.collection_count, search: .indices.search.query_time_in_millis}'
curl -s 'localhost:9200/_cat/thread_pool/search,write?v&h=node_name,name,active,queue,rejected'
curl -s localhost:9200/myindex/_stats/search?pretty | jq '.indices[].total.search'
SymptomUsual cause
rejected growing in a thread poolQueue full: too many concurrent requests, or shards too small and numerous
Old-generation GC frequentHeap pressure — field data, huge aggregations, or heap above 31 GB
Slow search, fast indexingToo many shards per node, or queries that cannot use filters
Slow indexingRefresh interval too low, replicas during bulk load, small bulk batches
Search results inconsistentRefresh has not run; the default is 1s, and refresh_interval may be longer

Shard sizing: aim for 10–50 GB per shard and no more than about 20 shards per GB of heap. Over-sharding is the most common self-inflicted performance problem.

# Slow log, per index
curl -XPUT localhost:9200/myindex/_settings -H 'content-type: application/json' -d '{
  "index.search.slowlog.threshold.query.warn": "2s",
  "index.indexing.slowlog.threshold.index.warn": "1s"
}'

Snapshots #

curl -XPUT localhost:9200/_snapshot/backups -H 'content-type: application/json' -d '{"type":"s3","settings":{"bucket":"es-backups","region":"ap-southeast-2"}}'
curl -XPUT "localhost:9200/_snapshot/backups/snap-$(date +%F)?wait_for_completion=false"
curl -s localhost:9200/_cat/snapshots/backups?v
curl -XPOST localhost:9200/_snapshot/backups/snap-2024-09-01/_restore -H 'content-type: application/json' -d '{"indices":"myindex","rename_pattern":"(.+)","rename_replacement":"restored-$1"}'

Oneliners #

# Cluster status in one line
curl -s localhost:9200/_cluster/health | jq -r '[.status, .number_of_nodes, .active_shards, .unassigned_shards] | @tsv'

# Indices sorted by size
curl -s 'localhost:9200/_cat/indices?h=index,store.size,docs.count&bytes=b' | sort -k2 -rn | head

# Unassigned shards with reasons
curl -s 'localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason' | awk '$4!="STARTED"'

# Retry failed allocations
curl -XPOST 'localhost:9200/_cluster/reroute?retry_failed=true'

# Disk use per node
curl -s 'localhost:9200/_cat/allocation?v&h=node,disk.percent,disk.used,disk.avail'

# Field mapping for one field across indices
curl -s 'localhost:9200/_mapping/field/service?pretty'

# What the analyser does to a string
curl -s localhost:9200/myindex/_analyze -H 'content-type: application/json' -d '{"field":"message","text":"Connection timed out"}' | jq -r '.tokens[].token'

# Count by a field, quickly
curl -s localhost:9200/myindex/_search -H 'content-type: application/json' -d '{"size":0,"aggs":{"s":{"terms":{"field":"service","size":20}}}}' | jq -r '.aggregations.s.buckets[] | [.key, .doc_count] | @tsv'

# Cancel a runaway search
curl -s 'localhost:9200/_tasks?actions=*search&detailed' | jq -r '.nodes[].tasks | to_entries[] | select(.value.running_time_in_nanos > 30e9) | .key' | xargs -I{} curl -XPOST "localhost:9200/_tasks/{}/_cancel"

# Documents matching a query, without fetching them
curl -s localhost:9200/myindex/_count -H 'content-type: application/json' -d '{"query":{"range":{"@timestamp":{"gte":"now-1h"}}}}' | jq .count

# Delete by query, then confirm
curl -s -XPOST 'localhost:9200/myindex/_delete_by_query?conflicts=proceed' -H 'content-type: application/json' -d '{"query":{"term":{"level":"debug"}}}' | jq '{deleted, failures}'

# Export a query's results as NDJSON
curl -s 'localhost:9200/myindex/_search?size=1000&scroll=1m' -H 'content-type: application/json' -d '{"query":{"match_all":{}}}' | jq -c '.hits.hits[]._source'

Last updated 15 September 2026 · Edit this page