Kubernetes
Kubernetes
kubectl for inspecting workloads, the object model behind them, and the checks that identify why a pod is not running.
Cheatsheet #
| Task | Command |
|---|---|
| Which cluster am I on | kubectl config current-context |
| Everything in a namespace | kubectl get all -n myns |
| Why is this pod unhappy | kubectl describe pod mypod -n myns |
| Events, newest last | kubectl get events -n myns --sort-by=.lastTimestamp |
| Previous container’s logs | kubectl logs mypod -c app --previous |
| Shell in a running pod | kubectl exec -it mypod -- sh |
| Debug a distroless pod | kubectl debug -it mypod --image=nicolaka/netshoot --target=app |
| Port-forward a service | kubectl port-forward svc/api 8080:80 -n myns |
| Restart a Deployment | kubectl rollout restart deploy/api -n myns |
| Watch a rollout | kubectl rollout status deploy/api -n myns |
| Roll back | kubectl rollout undo deploy/api -n myns |
| Scale | kubectl scale deploy/api --replicas=5 -n myns |
| Resource usage | kubectl top pod -n myns --sort-by=memory |
| Can I do this | kubectl auth can-i delete pods -n myns |
| Server-side dry run | kubectl apply -f x.yaml --dry-run=server |
| Raw object as stored | kubectl get pod mypod -o yaml |
Start with a failing workload #
Confirm the context, then read the pod’s state and its events. describe merges the object status with the events the scheduler and kubelet recorded, which is where the actual reason lives.
kubectl config current-context
kubectl get pods -n myns -o wide
kubectl describe pod mypod -n myns | sed -n '/Events:/,$p'
kubectl logs mypod -n myns -c app --previous --tail 100| Pod state | What it means | Next command |
|---|---|---|
Pending | No node fits: resources, taints, affinity, unbound PVC | kubectl describe pod and read the scheduler event |
ContainerCreating | Image pull, volume mount or CNI still working | kubectl describe pod, then kubelet logs on the node |
ImagePullBackOff | Wrong name, private registry, missing imagePullSecrets | kubectl get events, kubectl get sa default -o yaml |
CrashLoopBackOff | Process exits; kubelet backs off up to 5 minutes | kubectl logs --previous |
Running, not Ready | Readiness probe failing | kubectl describe pod, probe path and port |
OOMKilled (exit 137) | Working set exceeded limits.memory | kubectl top pod, raise the limit or fix the leak |
Terminating forever | Finalizer waiting, or terminationGracePeriodSeconds | kubectl get pod -o jsonpath='{.metadata.finalizers}' |
Evicted | Node pressure reclaimed it | kubectl describe node — look at conditions |
Changes in a GitOps-managed cluster
A manual edit is reverted by the reconciler. Apply persistent fixes through the repository that owns the resource; use direct commands for diagnosis only.
Core concepts #
The API server is the only component that writes to etcd. Everything else — controllers, schedulers, kubelets — watches the API for desired state and works to make reality match, so every fix is “change the desired state and wait”, never “make the change on the node”.
| Component | Job |
|---|---|
kube-apiserver | Validates and stores objects; the single write path |
etcd | Consistent key-value store holding cluster state |
kube-scheduler | Binds a pod to a node that satisfies its constraints |
kube-controller-manager | Reconciliation loops for Deployments, ReplicaSets, nodes, endpoints |
kubelet | Runs containers on one node and reports status |
kube-proxy or CNI replacement | Programs service load balancing on each node |
A Deployment owns a ReplicaSet, which owns Pods. Changing the pod template creates a new ReplicaSet and scales the old one down — that indirection is why kubectl rollout undo works and why deleting a pod does not fix a bad image.
kubectl #
kubectl config get-contexts
kubectl config use-context prod
kubectl config set-context --current --namespace=myns # stop typing -n
kubectl get deploy,sts,ds,job -A # workloads everywhere
kubectl get pods -A -o wide --field-selector status.phase!=Running
kubectl get pod mypod -o jsonpath='{.spec.containers[*].image}{"\n"}'
kubectl explain deployment.spec.strategy --recursive # schema, straight from the server
kubectl api-resources --namespaced=true # what kinds exist here
kubectl diff -f manifest.yaml # what applying would change
kubectl apply -f manifest.yaml --server-side # field ownership tracked by the serverkubectl get -o yaml returns the object after defaulting and admission, not what you submitted; kubectl.kubernetes.io/last-applied-configuration holds the client-side apply record.
Workloads #
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 3
revisionHistoryLimit: 5
strategy:
type: RollingUpdate
rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }
selector:
matchLabels: { app: api } # immutable after creation
template:
metadata:
labels: { app: api }
spec:
terminationGracePeriodSeconds: 45
securityContext:
runAsNonRoot: true
seccompProfile: { type: RuntimeDefault }
containers:
- name: app
image: registry.example.com/api@sha256:abc... # digest, not a moving tag
ports: [{ containerPort: 8080 }]
resources:
requests: { cpu: 100m, memory: 256Mi } # what the scheduler reserves
limits: { memory: 512Mi } # what the kernel enforces
readinessProbe:
httpGet: { path: /readyz, port: 8080 }
periodSeconds: 5
livenessProbe:
httpGet: { path: /healthz, port: 8080 }
initialDelaySeconds: 20
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }Requests drive scheduling and CPU shares; limits drive throttling and OOM kills. A CPU limit throttles rather than kills, which shows up as latency, not restarts — leaving CPU limits off and memory limits on is a common and defensible choice.
Readiness removes a pod from Service endpoints; liveness restarts the container. Pointing liveness at a dependency-checking endpoint turns a database blip into a cluster-wide restart storm.
kubectl rollout status deploy/api -n myns --timeout=5m
kubectl rollout history deploy/api -n myns
kubectl rollout undo deploy/api --to-revision=3 -n myns
kubectl rollout restart deploy/api -n myns # new pods, same spec: picks up rotated secrets
kubectl scale deploy/api --replicas=0 -n myns| Kind | Use it for |
|---|---|
Deployment | Stateless replicas, rolling updates, rollback |
StatefulSet | Stable network identity and per-replica storage; ordered, slow updates |
DaemonSet | One pod per node: agents, CNI, log shippers |
Job / CronJob | Run to completion, with backoffLimit and activeDeadlineSeconds |
A StatefulSet’s PVCs survive deletion of the StatefulSet by design; removing them is a separate, deliberate kubectl delete pvc.
Services and networking #
A Service is a stable virtual IP plus a selector. The endpoints controller keeps an EndpointSlice of ready pod IPs, and kube-proxy (or Cilium, or another replacement) programs the dataplane from it. No ready pods means no endpoints, which presents as connection refused rather than an error message.
kubectl get svc,endpointslice -n myns
kubectl get endpointslice -l kubernetes.io/service-name=api -n myns -o yaml | grep -A3 addresses
kubectl run tmp --rm -it --image=nicolaka/netshoot -- sh # curl, dig, tcpdump in-cluster
kubectl port-forward svc/api 8080:80 -n myns| Type | Behaviour |
|---|---|
ClusterIP | In-cluster virtual IP; the default |
NodePort | Same, plus a port on every node |
LoadBalancer | Same, plus a cloud load balancer provisioned by a controller |
ExternalName | CNAME only, no proxying |
headless (clusterIP: None) | DNS returns pod IPs directly; how StatefulSet members are addressed |
DNS names follow <service>.<namespace>.svc.cluster.local. Inside a pod, api resolves through search domains in /etc/resolv.conf; across namespaces, api.other-ns is the shortest reliable form.
NetworkPolicies are additive allow-lists: a pod selected by any policy denies everything not explicitly permitted, and a pod selected by none allows everything. Both an egress policy on the client and an ingress policy on the server must permit a flow.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: api-allow }
spec:
podSelector: { matchLabels: { app: api } }
policyTypes: [Ingress, Egress]
ingress:
- from:
- podSelector: { matchLabels: { app: web } }
ports: [{ protocol: TCP, port: 8080 }]
egress:
- to: [{ namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } } }]
ports: [{ protocol: UDP, port: 53 }] # forgetting DNS breaks everything elseStorage #
A PVC is a request; a PV is the volume. A StorageClass provisions PVs on demand, and its volumeBindingMode: WaitForFirstConsumer delays binding until a pod is scheduled so the volume lands in the right zone.
kubectl get pvc,pv -n myns
kubectl get sc
kubectl describe pvc data-api-0 -n myns # provisioning errors appear as eventsapiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: data }
spec:
accessModes: [ReadWriteOnce] # one node, not one pod
storageClassName: gp3
resources: { requests: { storage: 20Gi } }ReadWriteOnce allows many pods on the same node. Expansion works in place when the class sets allowVolumeExpansion: true; shrinking never does. A PVC stuck Terminating is usually still referenced by a running pod.
Configuration #
ConfigMaps and Secrets are the same mechanism with different handling: Secrets are base64-encoded in the API, encrypted at rest only if the cluster configures it, and mounted as tmpfs.
kubectl create configmap app-config --from-file=config.yaml --dry-run=client -o yaml > cm.yaml
kubectl create secret generic db --from-literal=password=... --dry-run=client -o yaml > secret.yaml
kubectl get secret db -o jsonpath='{.data.password}' | base64 -d envFrom:
- configMapRef: { name: app-config }
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef: { name: db, key: password }
volumeMounts:
- { name: config, mountPath: /etc/app, readOnly: true }
volumes:
- name: config
configMap: { name: app-config }Mounted ConfigMaps update in place within a minute or so; environment variables never do. Applications that read configuration once need kubectl rollout restart after a change.
Security #
Every pod runs as a ServiceAccount, whose token is projected into the pod and used for API calls. RBAC binds Roles (namespaced) or ClusterRoles (cluster-wide) to subjects; permissions are additive and there is no deny rule.
kubectl auth can-i --list -n myns # my permissions here
kubectl auth can-i get secrets -n myns --as system:serviceaccount:myns:api
kubectl get rolebinding,clusterrolebinding -A -o wide | grep mynskind: Role
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]Set automountServiceAccountToken: false on workloads that never call the API. Anything that can read Secrets in a namespace effectively owns that namespace.
Troubleshooting #
kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl describe node node-1 | sed -n '/Conditions:/,/Events:/p'
kubectl top node; kubectl top pod -A --sort-by=cpu
kubectl get pod mypod -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'
kubectl debug node/node-1 -it --image=busybox # host namespaces, for node-level checks
kubectl get --raw='/readyz?verbose' # API server health detail| Symptom | Likely cause |
|---|---|
| Pods pending, nodes look idle | Requests exceed allocatable, or taints without matching tolerations |
| Service intermittently fails | Some replicas failing readiness; check EndpointSlice membership |
| DNS resolution slow or failing | CoreDNS pods unhealthy, or NetworkPolicy blocking UDP 53 |
exec works, curl does not | Process bound to 127.0.0.1 inside the pod |
| Everything reverts after a minute | A GitOps controller owns the resource |
Node NotReady | kubelet stopped, disk pressure, or CNI failure — check node conditions |
Oneliners #
# Pods not Running or Succeeded, cluster-wide
kubectl get pods -A --field-selector 'status.phase!=Running,status.phase!=Succeeded'
# Top restart counts
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.containerStatuses[0].restartCount}{"\n"}{end}' | sort -k3 -nr | head
# Every image running in the cluster, deduplicated
kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort -u
# Requests and limits per pod
kubectl get pods -A -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,CPU:.spec.containers[*].resources.requests.cpu,MEM:.spec.containers[*].resources.requests.memory'
# Sum of CPU requests on a node
kubectl describe node node-1 | awk '/Allocated resources/,/Events/'
# Pods on one node
kubectl get pods -A -o wide --field-selector spec.nodeName=node-1
# Which pods mount a given secret
kubectl get pods -A -o json | jq -r '.items[] | select(.spec.volumes[]?.secret.secretName=="db") | "\(.metadata.namespace)/\(.metadata.name)"'
# Delete pods stuck Terminating (finalizer already cleared upstream)
kubectl delete pod mypod --grace-period=0 --force
# Drain a node for maintenance
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data --timeout=5m && kubectl uncordon node-1
# Watch rollout across every Deployment in a namespace
kubectl get deploy -n myns -o name | xargs -n1 -P0 kubectl rollout status -n myns
# API objects by count, to find what is filling etcd
kubectl get --raw=/metrics | grep '^apiserver_storage_objects' | sort -t' ' -k2 -nr | head
# Decode every key in a secret
kubectl get secret db -o go-template='{{range $k,$v := .data}}{{$k}}={{$v|base64decode}}{{"\n"}}{{end}}'
# Copy a file out of a pod without tar in the image
kubectl exec mypod -- cat /app/report.csv > report.csv