Data
Elasticsearch
Cluster and index health, query and aggregation syntax, mappings, and the checks for unassigned shards or a slow search.
Cheatsheet #
| Task | Request |
|---|---|
| Cluster health | GET _cluster/health?pretty |
| Why is it yellow or red | GET _cluster/allocation/explain |
| Indices by size | GET _cat/indices?v&s=store.size:desc |
| Shard placement | GET _cat/shards?v&s=state |
| Node resources | GET _cat/nodes?v&h=name,heap.percent,cpu,load_1m,disk.used_percent |
| Pending tasks | GET _cat/pending_tasks?v |
| Long-running tasks | GET _tasks?actions=*search&detailed |
| Mapping of an index | GET myindex/_mapping |
| Count matching | GET myindex/_count with a query |
| Explain a query’s score | GET myindex/_explain/<id> |
| Validate a query | GET myindex/_validate/query?explain |
| Analyse text | GET myindex/_analyze |
| Force a refresh | POST myindex/_refresh |
| Cluster settings | GET _cluster/settings?include_defaults&flat_settings |
Health and shards #
Green means every primary and replica is assigned; yellow means replicas are missing; red means a primary is missing and part of the data is unreadable. The allocation explain API states the exact reason rather than leaving it to inference.
curl -s localhost:9200/_cluster/health?pretty
curl -s 'localhost:9200/_cat/indices?v&health=red&s=index'
curl -s localhost:9200/_cluster/allocation/explain?pretty -H 'content-type: application/json' -d '{
"index": "logs-2024.09.01", "shard": 0, "primary": true
}'
curl -s 'localhost:9200/_cat/shards?v&s=state&h=index,shard,prirep,state,node,unassigned.reason'| Unassigned reason | Meaning |
|---|---|
INDEX_CREATED | Normal, briefly, after creation |
NODE_LEFT | A node disappeared; recovery in progress or blocked |
ALLOCATION_FAILED | Retries exhausted — POST _cluster/reroute?retry_failed=true |
NO_VALID_SHARD_COPY | Data is gone; only a restore or forced allocation recovers it |
DISK_THRESHOLD | Watermark exceeded; free space or adjust the watermark |
Disk watermarks (85% low, 90% high, 95% flood stage) move or freeze shards. The flood stage sets indices read-only, which presents as writes failing while the cluster looks green.
curl -s -XPUT localhost:9200/_cluster/settings -H 'content-type: application/json' -d '{
"persistent": {"cluster.routing.allocation.disk.watermark.flood_stage": "97%"}
}'
curl -s -XPUT 'localhost:9200/myindex/_settings' -H 'content-type: application/json' -d '{"index.blocks.read_only_allow_delete": null}'Indexing and mapping #
A mapping decides how text is analysed and therefore what can be searched. text is analysed into terms for full-text matching; keyword is stored whole for exact matching, sorting and aggregations. Most “why does my term query return nothing” questions are this distinction.
PUT myindex
{
"settings": { "number_of_shards": 1, "number_of_replicas": 1, "refresh_interval": "5s" },
"mappings": {
"dynamic": "strict",
"properties": {
"service": { "type": "keyword" },
"message": { "type": "text", "analyzer": "standard" },
"level": { "type": "keyword" },
"duration_ms": { "type": "float" },
"@timestamp": { "type": "date" },
"host": {
"properties": { "name": { "type": "keyword" }, "ip": { "type": "ip" } }
}
}
}
}Mappings are immutable for existing fields: changing a type requires a new index and _reindex. dynamic: strict rejects unexpected fields instead of guessing a type and locking it in.
curl -XPOST localhost:9200/myindex/_doc -H 'content-type: application/json' -d '{"service":"api","message":"timeout"}'
curl -XPUT localhost:9200/myindex/_doc/1 -H 'content-type: application/json' -d '{"service":"api"}'
curl -XPOST localhost:9200/myindex/_update/1 -H 'content-type: application/json' -d '{"doc":{"level":"error"}}'
curl -XPOST localhost:9200/_bulk -H 'content-type: application/x-ndjson' --data-binary @bulk.ndjson{"index":{"_index":"myindex","_id":"1"}}
{"service":"api","message":"timeout","@timestamp":"2024-09-01T00:00:00Z"}Bulk is the only sane way to index volume: one request per document wastes a round trip and a refresh cycle each time. Newlines matter, including the trailing one.
Searching #
GET myindex/_search
{
"size": 20,
"track_total_hits": true,
"query": {
"bool": {
"must": [{ "match": { "message": "connection timeout" } }],
"filter": [
{ "term": { "service": "api" } },
{ "range": { "@timestamp": { "gte": "now-1h" } } }
],
"must_not": [{ "term": { "level": "debug" } }],
"should": [{ "match_phrase": { "message": "connection refused" } }]
}
},
"sort": [{ "@timestamp": "desc" }],
"_source": ["service", "message", "@timestamp"]
}| Clause | Effect |
|---|---|
must | Must match, contributes to the score |
filter | Must match, no score, cacheable — use it for everything structured |
should | Boosts when matched |
must_not | Excludes, no score |
match | Analysed full-text search |
term | Exact, unanalysed — on a text field it will usually match nothing |
match_phrase | Terms in order |
wildcard / regexp | Expensive; avoid leading wildcards entirely |
Put date ranges and identifiers in filter, not must: filters skip scoring and are cached per segment, which is often an order of magnitude cheaper.
GET myindex/_search
{
"size": 0,
"aggs": {
"by_service": {
"terms": { "field": "service", "size": 10 },
"aggs": {
"p95": { "percentiles": { "field": "duration_ms", "percents": [95] } },
"over_time": { "date_histogram": { "field": "@timestamp", "fixed_interval": "5m" } }
}
}
}
}size: 0 skips returning documents when only aggregations are wanted. terms aggregations on high-cardinality fields are memory-hungry — use composite for pagination over many buckets.
Reindex, aliases and lifecycle #
Aliases let an index be replaced without changing the client. Write to an alias, reindex into a new index, then swap atomically.
curl -XPOST localhost:9200/_reindex -H 'content-type: application/json' -d '{
"source": {"index": "myindex"}, "dest": {"index": "myindex-v2"}
}'
curl -XPOST localhost:9200/_aliases -H 'content-type: application/json' -d '{
"actions": [
{"remove": {"index": "myindex", "alias": "myindex-current"}},
{"add": {"index": "myindex-v2", "alias": "myindex-current"}}
]
}'For time-series data use data streams with an ILM policy: hot for writes, warm for search, delete at a fixed age. Nothing should be deleting log indices with a cron job and a date pattern.
curl -s localhost:9200/_ilm/policy/logs?pretty
curl -s 'localhost:9200/_cat/indices/logs-*?v&s=index' | tail
curl -s localhost:9200/myindex/_ilm/explain?prettyPerformance #
curl -s 'localhost:9200/_nodes/stats/jvm,indices?pretty' | jq '.nodes[] | {heap: .jvm.mem.heap_used_percent, gc: .jvm.gc.collectors.old.collection_count, search: .indices.search.query_time_in_millis}'
curl -s 'localhost:9200/_cat/thread_pool/search,write?v&h=node_name,name,active,queue,rejected'
curl -s localhost:9200/myindex/_stats/search?pretty | jq '.indices[].total.search'| Symptom | Usual cause |
|---|---|
rejected growing in a thread pool | Queue full: too many concurrent requests, or shards too small and numerous |
| Old-generation GC frequent | Heap pressure — field data, huge aggregations, or heap above 31 GB |
| Slow search, fast indexing | Too many shards per node, or queries that cannot use filters |
| Slow indexing | Refresh interval too low, replicas during bulk load, small bulk batches |
| Search results inconsistent | Refresh has not run; the default is 1s, and refresh_interval may be longer |
Shard sizing: aim for 10–50 GB per shard and no more than about 20 shards per GB of heap. Over-sharding is the most common self-inflicted performance problem.
# Slow log, per index
curl -XPUT localhost:9200/myindex/_settings -H 'content-type: application/json' -d '{
"index.search.slowlog.threshold.query.warn": "2s",
"index.indexing.slowlog.threshold.index.warn": "1s"
}'Snapshots #
curl -XPUT localhost:9200/_snapshot/backups -H 'content-type: application/json' -d '{"type":"s3","settings":{"bucket":"es-backups","region":"ap-southeast-2"}}'
curl -XPUT "localhost:9200/_snapshot/backups/snap-$(date +%F)?wait_for_completion=false"
curl -s localhost:9200/_cat/snapshots/backups?v
curl -XPOST localhost:9200/_snapshot/backups/snap-2024-09-01/_restore -H 'content-type: application/json' -d '{"indices":"myindex","rename_pattern":"(.+)","rename_replacement":"restored-$1"}'Oneliners #
# Cluster status in one line
curl -s localhost:9200/_cluster/health | jq -r '[.status, .number_of_nodes, .active_shards, .unassigned_shards] | @tsv'
# Indices sorted by size
curl -s 'localhost:9200/_cat/indices?h=index,store.size,docs.count&bytes=b' | sort -k2 -rn | head
# Unassigned shards with reasons
curl -s 'localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason' | awk '$4!="STARTED"'
# Retry failed allocations
curl -XPOST 'localhost:9200/_cluster/reroute?retry_failed=true'
# Disk use per node
curl -s 'localhost:9200/_cat/allocation?v&h=node,disk.percent,disk.used,disk.avail'
# Field mapping for one field across indices
curl -s 'localhost:9200/_mapping/field/service?pretty'
# What the analyser does to a string
curl -s localhost:9200/myindex/_analyze -H 'content-type: application/json' -d '{"field":"message","text":"Connection timed out"}' | jq -r '.tokens[].token'
# Count by a field, quickly
curl -s localhost:9200/myindex/_search -H 'content-type: application/json' -d '{"size":0,"aggs":{"s":{"terms":{"field":"service","size":20}}}}' | jq -r '.aggregations.s.buckets[] | [.key, .doc_count] | @tsv'
# Cancel a runaway search
curl -s 'localhost:9200/_tasks?actions=*search&detailed' | jq -r '.nodes[].tasks | to_entries[] | select(.value.running_time_in_nanos > 30e9) | .key' | xargs -I{} curl -XPOST "localhost:9200/_tasks/{}/_cancel"
# Documents matching a query, without fetching them
curl -s localhost:9200/myindex/_count -H 'content-type: application/json' -d '{"query":{"range":{"@timestamp":{"gte":"now-1h"}}}}' | jq .count
# Delete by query, then confirm
curl -s -XPOST 'localhost:9200/myindex/_delete_by_query?conflicts=proceed' -H 'content-type: application/json' -d '{"query":{"term":{"level":"debug"}}}' | jq '{deleted, failures}'
# Export a query's results as NDJSON
curl -s 'localhost:9200/myindex/_search?size=1000&scroll=1m' -H 'content-type: application/json' -d '{"query":{"match_all":{}}}' | jq -c '.hits.hits[]._source'