Software Engineering Wiki

Kubernetes

Cilium

Identity-based policy, the eBPF datapath, Hubble observability and the commands for tracing a dropped flow.

Cheatsheet #

TaskCommand
Overall healthcilium status --wait
Agent detail on one nodekubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium-dbg status --verbose
Endpoints on a nodekubectl exec -n kube-system ds/cilium -- cilium-dbg endpoint list
Identity of a podkubectl exec -n kube-system ds/cilium -- cilium-dbg identity list
Watch drops livekubectl exec -n kube-system ds/cilium -- cilium-dbg monitor --type drop
Flows for a podhubble observe --pod myns/api --last 50
Only denied flowshubble observe --verdict DROPPED --last 100
Policy verdict for a flowhubble observe --to-pod myns/api --verdict DROPPED -o json | jq .drop_reason_desc
Services programmed in eBPFkubectl exec -n kube-system ds/cilium -- cilium-dbg service list
Is kube-proxy replacedkubectl exec -n kube-system ds/cilium -- cilium-dbg status | grep KubeProxyReplacement
End-to-end testcilium connectivity test
Policy applied to an endpointcilium-dbg endpoint get <id> -o json | jq .status.policy

How Cilium sees the cluster #

Cilium attaches eBPF programs to the kernel networking hooks on each node, so packets are handled in the kernel rather than by iptables chains. Every pod becomes an endpoint with a numeric identity derived from its labels, and policy is enforced on identity, not IP address.

That indirection is the important part: a pod rescheduled to another node keeps its identity, so policy does not churn when IPs change, and a policy decision is a map lookup rather than a linear walk of rules.

cilium status --wait                                   # CLI, from outside the cluster
kubectl exec -n kube-system ds/cilium -- cilium-dbg status --verbose   # per-node agent view
kubectl exec -n kube-system ds/cilium -- cilium-dbg endpoint list      # identity per local pod
kubectl exec -n kube-system ds/cilium -- cilium-dbg identity list      # identity to label mapping

Reserved identities matter when reading verdicts: reserved:host, reserved:remote-node, reserved:world (anything outside the cluster), reserved:kube-apiserver, and reserved:unmanaged.

Datapath and IPAM #

ModeHow packets leave the nodeWhen it applies
Encapsulation (VXLAN/Geneve)Wrapped in a tunnel between nodesDefault; works on any underlay
Native routingRouted by the underlay using pod CIDRsUnderlay already knows the pod routes
ENI / Azure / GKE IPAMPod gets a real VPC addressCloud-native addressing, no tunnel overhead
kubectl exec -n kube-system ds/cilium -- cilium-dbg status | grep -E 'Routing|IPAM|Masquerading'
kubectl get ciliumnode -o custom-columns='NODE:.metadata.name,CIDR:.spec.ipam.podCIDRs,USED:.status.ipam.used'
kubectl -n kube-system get cm cilium-config -o yaml | grep -E 'tunnel|routing-mode|ipam|masquerade'

Address exhaustion shows as pods stuck in ContainerCreating with CNI errors; compare used against the node’s allocation in CiliumNode.

kube-proxy replacement #

With kubeProxyReplacement=true Cilium programs service load balancing in eBPF at the socket and TC layers, so a ClusterIP connection is translated before it reaches the network stack. iptables -t nat -L shows nothing useful on such a cluster — service state lives in eBPF maps.

kubectl exec -n kube-system ds/cilium -- cilium-dbg status | grep KubeProxyReplacement
kubectl exec -n kube-system ds/cilium -- cilium-dbg service list
kubectl exec -n kube-system ds/cilium -- cilium-dbg bpf lb list
kubectl exec -n kube-system ds/cilium -- cilium-dbg bpf ct list global | head

DSR returns responses straight from the backend node to the client, preserving the source IP and halving the return path; Maglev gives consistent backend selection across nodes so that a node restart does not reshuffle existing flows. Both are set in the Helm values (loadBalancer.mode, loadBalancer.algorithm) and need native routing or a supporting underlay.

Network policy #

Kubernetes NetworkPolicy works unchanged. CiliumNetworkPolicy adds L7 rules, DNS-based egress, entity selectors and cluster-wide scope. Policies are allow-lists: once any policy selects an endpoint, that direction defaults to deny.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata: { name: api, namespace: apps }
spec:
  endpointSelector:
    matchLabels: { app: api }
  ingress:
    - fromEndpoints: [{ matchLabels: { app: web } }]
      toPorts:
        - ports: [{ port: "8080", protocol: TCP }]
          rules:
            http:
              - { method: "GET", path: "/v1/.*" }      # L7: enforced by the Envoy proxy
  egress:
    - toEndpoints: [{ matchLabels: { "k8s:io.kubernetes.pod.namespace": kube-system, "k8s:k8s-app": kube-dns } }]
      toPorts:
        - ports: [{ port: "53", protocol: UDP }]
          rules:
            dns: [{ matchPattern: "*" }]                # required for toFQDNs to work
    - toFQDNs: [{ matchName: "api.vendor.example.com" }]
      toPorts: [{ ports: [{ port: "443", protocol: TCP }] }]
    - toEntities: ["kube-apiserver"]

L7 rules redirect matching traffic through Envoy, so an L7 policy changes the datapath and the latency profile — apply it where it earns that cost. toFQDNs works by snooping DNS responses, which is why the DNS egress rule with rules.dns must exist first.

CiliumClusterwideNetworkPolicy is the same schema without a namespace, for baseline rules such as “everything may reach CoreDNS, nothing may reach the metadata service”. Host policies (nodeSelector with reserved:host) protect the node itself and can lock you out of SSH if the rule set omits it.

kubectl get cnp,ccnp -A
kubectl exec -n kube-system ds/cilium -- cilium-dbg policy get
kubectl exec -n kube-system ds/cilium -- cilium-dbg endpoint get <id> -o json | jq '.status.policy.realized'

Hubble #

Hubble reads flow events from the same eBPF programs that enforce policy, so a verdict in Hubble is the verdict the datapath applied — not a reconstruction.

cilium hubble enable --ui
cilium hubble port-forward &                       # exposes the relay locally
hubble status
hubble observe --pod apps/api --last 50
hubble observe --verdict DROPPED --last 100
hubble observe --to-pod apps/api --port 8080 -f
hubble observe --protocol dns --last 20
hubble observe -o json | jq -r 'select(.verdict=="DROPPED") | [.source.pod_name, .destination.pod_name, .drop_reason_desc] | @tsv'
Drop reasonMeaning
POLICY_DENIEDNo allow rule for this identity pair and port
CT_TRUNCATED / CT_MAP_INSERTION_FAILEDConnection tracking table pressure
UNSUPPORTED_L3_PROTOCOLNon-IP traffic on a managed interface
FRAG_NEEDEDMTU mismatch, typically tunnel overhead not accounted for
NO_SERVICE_BACKENDService exists with zero ready endpoints

Troubleshooting #

cilium status --wait
cilium connectivity test --test-namespace cilium-test    # full end-to-end suite, creates pods
kubectl exec -n kube-system ds/cilium -- cilium-dbg monitor --type drop --type trace
kubectl exec -n kube-system ds/cilium -- cilium-dbg bpf tunnel list
kubectl exec -n kube-system ds/cilium -- cilium-dbg map list --verbose
cilium sysdump                                            # bundle for support: logs, maps, policies
SymptomWhere to look
Pods cannot resolve DNSDNS egress policy, CoreDNS endpoints, hubble observe --protocol dns
Intermittent loss between nodesMTU: tunnel overhead versus underlay MTU (cilium-dbg status | grep MTU)
Service IP unreachablecilium-dbg service list, then whether backends are ready
Policy has no effectEndpoint selector labels do not match; check cilium-dbg endpoint list labels
Agent restarts repeatedlyKernel version or missing eBPF features — cilium-dbg status --verbose prints probes
Traffic allowed that should not beA broader CiliumClusterwideNetworkPolicy or a reserved: entity rule

Host policies can remove your access

A CiliumClusterwideNetworkPolicy selecting reserved:host without an explicit SSH and kubelet allow rule locks the node. Stage it on one node with console access before rolling it out.

Oneliners #

# Identity for every pod on this node
kubectl exec -n kube-system ds/cilium -- cilium-dbg endpoint list -o json | jq -r '.[] | [.status.identity.id, .status.external-identifiers.k8s-namespace, .status.external-identifiers.k8s-pod-name] | @tsv'

# Which agents are unhealthy
kubectl get pods -n kube-system -l k8s-app=cilium -o wide | grep -v Running

# Drops per source pod in the last minutes
hubble observe --verdict DROPPED --last 500 -o json | jq -r '.source.pod_name' | sort | uniq -c | sort -rn

# Top talkers by flow count
hubble observe --last 1000 -o json | jq -r '[.source.pod_name, .destination.pod_name] | @tsv' | sort | uniq -c | sort -rn | head

# Confirm a service has backends in the datapath
kubectl exec -n kube-system ds/cilium -- cilium-dbg service list | grep -A2 10.96.0.1

# Every FQDN policy currently resolved
kubectl exec -n kube-system ds/cilium -- cilium-dbg fqdn cache list | head -20

# Compare policy revision across agents
kubectl get pods -n kube-system -l k8s-app=cilium -o name | xargs -I{} sh -c 'echo -n "{} "; kubectl exec -n kube-system {} -c cilium-agent -- cilium-dbg policy get -o json | jq .revision'

# Connectivity test without leaving resources behind
cilium connectivity test --test-namespace cilium-test && kubectl delete ns cilium-test

# MTU actually in use
kubectl exec -n kube-system ds/cilium -- cilium-dbg status --verbose | grep -i mtu

Last updated 15 September 2026 · Edit this page