Software Engineering Wiki

Kubernetes

Kubernetes

kubectl for inspecting workloads, the object model behind them, and the checks that identify why a pod is not running.

Cheatsheet #

TaskCommand
Which cluster am I onkubectl config current-context
Everything in a namespacekubectl get all -n myns
Why is this pod unhappykubectl describe pod mypod -n myns
Events, newest lastkubectl get events -n myns --sort-by=.lastTimestamp
Previous container’s logskubectl logs mypod -c app --previous
Shell in a running podkubectl exec -it mypod -- sh
Debug a distroless podkubectl debug -it mypod --image=nicolaka/netshoot --target=app
Port-forward a servicekubectl port-forward svc/api 8080:80 -n myns
Restart a Deploymentkubectl rollout restart deploy/api -n myns
Watch a rolloutkubectl rollout status deploy/api -n myns
Roll backkubectl rollout undo deploy/api -n myns
Scalekubectl scale deploy/api --replicas=5 -n myns
Resource usagekubectl top pod -n myns --sort-by=memory
Can I do thiskubectl auth can-i delete pods -n myns
Server-side dry runkubectl apply -f x.yaml --dry-run=server
Raw object as storedkubectl get pod mypod -o yaml

Start with a failing workload #

Confirm the context, then read the pod’s state and its events. describe merges the object status with the events the scheduler and kubelet recorded, which is where the actual reason lives.

kubectl config current-context
kubectl get pods -n myns -o wide
kubectl describe pod mypod -n myns | sed -n '/Events:/,$p'
kubectl logs mypod -n myns -c app --previous --tail 100
Pod stateWhat it meansNext command
PendingNo node fits: resources, taints, affinity, unbound PVCkubectl describe pod and read the scheduler event
ContainerCreatingImage pull, volume mount or CNI still workingkubectl describe pod, then kubelet logs on the node
ImagePullBackOffWrong name, private registry, missing imagePullSecretskubectl get events, kubectl get sa default -o yaml
CrashLoopBackOffProcess exits; kubelet backs off up to 5 minuteskubectl logs --previous
Running, not ReadyReadiness probe failingkubectl describe pod, probe path and port
OOMKilled (exit 137)Working set exceeded limits.memorykubectl top pod, raise the limit or fix the leak
Terminating foreverFinalizer waiting, or terminationGracePeriodSecondskubectl get pod -o jsonpath='{.metadata.finalizers}'
EvictedNode pressure reclaimed itkubectl describe node — look at conditions

Changes in a GitOps-managed cluster

A manual edit is reverted by the reconciler. Apply persistent fixes through the repository that owns the resource; use direct commands for diagnosis only.

Core concepts #

The API server is the only component that writes to etcd. Everything else — controllers, schedulers, kubelets — watches the API for desired state and works to make reality match, so every fix is “change the desired state and wait”, never “make the change on the node”.

ComponentJob
kube-apiserverValidates and stores objects; the single write path
etcdConsistent key-value store holding cluster state
kube-schedulerBinds a pod to a node that satisfies its constraints
kube-controller-managerReconciliation loops for Deployments, ReplicaSets, nodes, endpoints
kubeletRuns containers on one node and reports status
kube-proxy or CNI replacementPrograms service load balancing on each node

A Deployment owns a ReplicaSet, which owns Pods. Changing the pod template creates a new ReplicaSet and scales the old one down — that indirection is why kubectl rollout undo works and why deleting a pod does not fix a bad image.

kubectl #

kubectl config get-contexts
kubectl config use-context prod
kubectl config set-context --current --namespace=myns   # stop typing -n

kubectl get deploy,sts,ds,job -A                         # workloads everywhere
kubectl get pods -A -o wide --field-selector status.phase!=Running
kubectl get pod mypod -o jsonpath='{.spec.containers[*].image}{"\n"}'
kubectl explain deployment.spec.strategy --recursive      # schema, straight from the server
kubectl api-resources --namespaced=true                   # what kinds exist here
kubectl diff -f manifest.yaml                             # what applying would change
kubectl apply -f manifest.yaml --server-side              # field ownership tracked by the server

kubectl get -o yaml returns the object after defaulting and admission, not what you submitted; kubectl.kubernetes.io/last-applied-configuration holds the client-side apply record.

Workloads #

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  revisionHistoryLimit: 5
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
  selector:
    matchLabels: { app: api }              # immutable after creation
  template:
    metadata:
      labels: { app: api }
    spec:
      terminationGracePeriodSeconds: 45
      securityContext:
        runAsNonRoot: true
        seccompProfile: { type: RuntimeDefault }
      containers:
        - name: app
          image: registry.example.com/api@sha256:abc...   # digest, not a moving tag
          ports: [{ containerPort: 8080 }]
          resources:
            requests: { cpu: 100m, memory: 256Mi }        # what the scheduler reserves
            limits: { memory: 512Mi }                     # what the kernel enforces
          readinessProbe:
            httpGet: { path: /readyz, port: 8080 }
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 20
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }

Requests drive scheduling and CPU shares; limits drive throttling and OOM kills. A CPU limit throttles rather than kills, which shows up as latency, not restarts — leaving CPU limits off and memory limits on is a common and defensible choice.

Readiness removes a pod from Service endpoints; liveness restarts the container. Pointing liveness at a dependency-checking endpoint turns a database blip into a cluster-wide restart storm.

kubectl rollout status deploy/api -n myns --timeout=5m
kubectl rollout history deploy/api -n myns
kubectl rollout undo deploy/api --to-revision=3 -n myns
kubectl rollout restart deploy/api -n myns       # new pods, same spec: picks up rotated secrets
kubectl scale deploy/api --replicas=0 -n myns
KindUse it for
DeploymentStateless replicas, rolling updates, rollback
StatefulSetStable network identity and per-replica storage; ordered, slow updates
DaemonSetOne pod per node: agents, CNI, log shippers
Job / CronJobRun to completion, with backoffLimit and activeDeadlineSeconds

A StatefulSet’s PVCs survive deletion of the StatefulSet by design; removing them is a separate, deliberate kubectl delete pvc.

Services and networking #

A Service is a stable virtual IP plus a selector. The endpoints controller keeps an EndpointSlice of ready pod IPs, and kube-proxy (or Cilium, or another replacement) programs the dataplane from it. No ready pods means no endpoints, which presents as connection refused rather than an error message.

kubectl get svc,endpointslice -n myns
kubectl get endpointslice -l kubernetes.io/service-name=api -n myns -o yaml | grep -A3 addresses
kubectl run tmp --rm -it --image=nicolaka/netshoot -- sh   # curl, dig, tcpdump in-cluster
kubectl port-forward svc/api 8080:80 -n myns
TypeBehaviour
ClusterIPIn-cluster virtual IP; the default
NodePortSame, plus a port on every node
LoadBalancerSame, plus a cloud load balancer provisioned by a controller
ExternalNameCNAME only, no proxying
headless (clusterIP: None)DNS returns pod IPs directly; how StatefulSet members are addressed

DNS names follow <service>.<namespace>.svc.cluster.local. Inside a pod, api resolves through search domains in /etc/resolv.conf; across namespaces, api.other-ns is the shortest reliable form.

NetworkPolicies are additive allow-lists: a pod selected by any policy denies everything not explicitly permitted, and a pod selected by none allows everything. Both an egress policy on the client and an ingress policy on the server must permit a flow.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: api-allow }
spec:
  podSelector: { matchLabels: { app: api } }
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - podSelector: { matchLabels: { app: web } }
      ports: [{ protocol: TCP, port: 8080 }]
  egress:
    - to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } } }]
      ports: [{ protocol: UDP, port: 53 }]   # forgetting DNS breaks everything else

Storage #

A PVC is a request; a PV is the volume. A StorageClass provisions PVs on demand, and its volumeBindingMode: WaitForFirstConsumer delays binding until a pod is scheduled so the volume lands in the right zone.

kubectl get pvc,pv -n myns
kubectl get sc
kubectl describe pvc data-api-0 -n myns      # provisioning errors appear as events
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: data }
spec:
  accessModes: [ReadWriteOnce]               # one node, not one pod
  storageClassName: gp3
  resources: { requests: { storage: 20Gi } }

ReadWriteOnce allows many pods on the same node. Expansion works in place when the class sets allowVolumeExpansion: true; shrinking never does. A PVC stuck Terminating is usually still referenced by a running pod.

Configuration #

ConfigMaps and Secrets are the same mechanism with different handling: Secrets are base64-encoded in the API, encrypted at rest only if the cluster configures it, and mounted as tmpfs.

kubectl create configmap app-config --from-file=config.yaml --dry-run=client -o yaml > cm.yaml
kubectl create secret generic db --from-literal=password=... --dry-run=client -o yaml > secret.yaml
kubectl get secret db -o jsonpath='{.data.password}' | base64 -d
    envFrom:
      - configMapRef: { name: app-config }
    env:
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef: { name: db, key: password }
    volumeMounts:
      - { name: config, mountPath: /etc/app, readOnly: true }
  volumes:
    - name: config
      configMap: { name: app-config }

Mounted ConfigMaps update in place within a minute or so; environment variables never do. Applications that read configuration once need kubectl rollout restart after a change.

Security #

Every pod runs as a ServiceAccount, whose token is projected into the pod and used for API calls. RBAC binds Roles (namespaced) or ClusterRoles (cluster-wide) to subjects; permissions are additive and there is no deny rule.

kubectl auth can-i --list -n myns                                   # my permissions here
kubectl auth can-i get secrets -n myns --as system:serviceaccount:myns:api
kubectl get rolebinding,clusterrolebinding -A -o wide | grep myns
kind: Role
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]

Set automountServiceAccountToken: false on workloads that never call the API. Anything that can read Secrets in a namespace effectively owns that namespace.

Troubleshooting #

kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl describe node node-1 | sed -n '/Conditions:/,/Events:/p'
kubectl top node; kubectl top pod -A --sort-by=cpu
kubectl get pod mypod -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'
kubectl debug node/node-1 -it --image=busybox     # host namespaces, for node-level checks
kubectl get --raw='/readyz?verbose'               # API server health detail
SymptomLikely cause
Pods pending, nodes look idleRequests exceed allocatable, or taints without matching tolerations
Service intermittently failsSome replicas failing readiness; check EndpointSlice membership
DNS resolution slow or failingCoreDNS pods unhealthy, or NetworkPolicy blocking UDP 53
exec works, curl does notProcess bound to 127.0.0.1 inside the pod
Everything reverts after a minuteA GitOps controller owns the resource
Node NotReadykubelet stopped, disk pressure, or CNI failure — check node conditions

Oneliners #

# Pods not Running or Succeeded, cluster-wide
kubectl get pods -A --field-selector 'status.phase!=Running,status.phase!=Succeeded'

# Top restart counts
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.containerStatuses[0].restartCount}{"\n"}{end}' | sort -k3 -nr | head

# Every image running in the cluster, deduplicated
kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort -u

# Requests and limits per pod
kubectl get pods -A -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,CPU:.spec.containers[*].resources.requests.cpu,MEM:.spec.containers[*].resources.requests.memory'

# Sum of CPU requests on a node
kubectl describe node node-1 | awk '/Allocated resources/,/Events/'

# Pods on one node
kubectl get pods -A -o wide --field-selector spec.nodeName=node-1

# Which pods mount a given secret
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.volumes[]?.secret.secretName=="db") | "\(.metadata.namespace)/\(.metadata.name)"'

# Delete pods stuck Terminating (finalizer already cleared upstream)
kubectl delete pod mypod --grace-period=0 --force

# Drain a node for maintenance
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data --timeout=5m && kubectl uncordon node-1

# Watch rollout across every Deployment in a namespace
kubectl get deploy -n myns -o name | xargs -n1 -P0 kubectl rollout status -n myns

# API objects by count, to find what is filling etcd
kubectl get --raw=/metrics | grep '^apiserver_storage_objects' | sort -t' ' -k2 -nr | head

# Decode every key in a secret
kubectl get secret db -o go-template='{{range $k,$v := .data}}{{$k}}={{$v|base64decode}}{{"\n"}}{{end}}'

# Copy a file out of a pod without tar in the image
kubectl exec mypod -- cat /app/report.csv > report.csv

Last updated 15 September 2026 · Edit this page