OpenTelemetry · Software Engineering Wiki

Cheatsheet #

Task	Command or setting
Point an SDK at a collector	`OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317`
Name the service	`OTEL_SERVICE_NAME=api` (or `service.name` in `OTEL_RESOURCE_ATTRIBUTES`)
Sample 10% head-based	`OTEL_TRACES_SAMPLER=parentbased_traceidratio OTEL_TRACES_SAMPLER_ARG=0.1`
Turn off a signal	`OTEL_METRICS_EXPORTER=none`
Validate collector config	`otelcol validate --config config.yaml`
See what the collector receives	add the `debug` exporter with `verbosity: detailed`
Collector’s own metrics	`curl localhost:8888/metrics`
Health endpoint	`curl localhost:13133`
Pipeline internals	`curl localhost:55679/debug/tracez` (zpages extension)
Trace headers on the wire	`curl -v` and look for `traceparent`

The model #

Three signals share one resource and one context. A trace is a tree of spans with a shared trace ID; a metric is an aggregated measurement; a log is a timestamped record. Attaching the trace ID to metrics (exemplars) and logs is what makes the three navigable as one story.

A resource describes the producer — service.name, service.version, deployment.environment, k8s.pod.name — and is attached to every signal from that process. Getting service.name wrong makes everything else unusable, because every backend groups by it.

Context propagation carries the trace ID across process boundaries in the traceparent header (W3C Trace Context). A trace breaks exactly where propagation breaks: an uninstrumented hop, a queue that drops headers, or a mismatched propagator between services.

Collector configuration #

The Collector is receivers → processors → exporters, wired into named pipelines. Processors run in the order listed, which is the only ordering guarantee that matters.

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
  prometheus:
    config:
      scrape_configs:
        - job_name: apps
          kubernetes_sd_configs: [{ role: pod }]

processors:
  memory_limiter:                 # must be first: sheds load before the process is OOM killed
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  k8sattributes:                  # adds pod, namespace, node and workload attributes
    auth_type: serviceAccount
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.node.name]
  resourcedetection:
    detectors: [env, system, eks]
  transform:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          - delete_key(attributes, "http.request.header.authorization")
  batch:                          # must be last: amortises export cost
    timeout: 5s
    send_batch_size: 8192

exporters:
  otlphttp/tempo:
    endpoint: https://tempo.example.internal
    retry_on_failure: { enabled: true, max_elapsed_time: 300s }
    sending_queue: { enabled: true, queue_size: 5000 }
  prometheusremotewrite:
    endpoint: https://mimir.example.internal/api/v1/push
  debug:
    verbosity: detailed           # troubleshooting only, it prints every record

extensions:
  health_check: { endpoint: 0.0.0.0:13133 }
  pprof: {}
  zpages: {}

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, transform, batch]
      exporters: [otlphttp/tempo]
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [prometheusremotewrite]
  telemetry:
    metrics: { level: detailed, address: 0.0.0.0:8888 }

memory_limiter first and batch last is not style — a limiter after batching cannot shed load in time, and batching before sampling wastes the work it was meant to save.

Deployment shape #

Shape	Use
Agent (DaemonSet)	Receives from local pods, adds node and pod attributes, forwards
Gateway (Deployment)	Central pipeline: tail sampling, redaction, fan-out to backends
Sidecar	Only when a workload needs an isolated pipeline or its own credentials

Agent plus gateway is the default for Kubernetes: the agent is the only thing that can attach node-local metadata, the gateway is the only place tail sampling can see all spans of a trace.

Instrumentation #

OTEL_SERVICE_NAME=api
OTEL_RESOURCE_ATTRIBUTES=service.version=1.4.2,deployment.environment=prod
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1
OTEL_PROPAGATORS=tracecontext,baggage

Auto-instrumentation covers the frameworks and clients; manual spans are for business operations the framework cannot name.

tr := otel.Tracer("checkout")
ctx, span := tr.Start(ctx, "reserve_inventory")
defer span.End()
span.SetAttributes(attribute.String("sku", sku), attribute.Int("qty", qty))
if err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, "reservation failed")
}

Span attributes are as cardinality-sensitive as metric labels at the backend, but unlike metrics a high-cardinality attribute here is usually correct — that is what traces are for. Never put credentials, tokens or personal data in attributes; redact in the Collector if the SDK cannot.

The Operator injects SDKs without changing images:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata: { name: default }
spec:
  exporter: { endpoint: http://otel-collector:4317 }
  propagators: [tracecontext, baggage]
  sampler: { type: parentbased_traceidratio, argument: "0.1" }

# pod annotation
instrumentation.opentelemetry.io/inject-java: "true"

Sampling #

Head sampling decides at the root span and is cheap but blind: it cannot know the request will fail. Tail sampling buffers a whole trace in the gateway and decides afterwards, which is the only way to keep every error and slow request.

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

Every span of a trace must reach the same tail-sampling instance, so scale the gateway with a load balancing exporter keyed on trace ID, not a plain round robin.

decision_wait must exceed your longest trace, or late spans arrive after the decision and are dropped, producing traces that look truncated at the slow service — exactly the one you were trying to keep.

Oneliners #

# Validate before rolling out
otelcol validate --config config.yaml

# What the collector is dropping and why
curl -s localhost:8888/metrics | grep -E 'otelcol_(processor_dropped|exporter_send_failed|receiver_refused)'

# Queue saturation
curl -s localhost:8888/metrics | grep -E 'otelcol_exporter_queue_(size|capacity)'

# Send a test span over OTLP/HTTP
curl -X POST http://localhost:4318/v1/traces -H 'Content-Type: application/json' -d @span.json

# Confirm propagation reaches a service
curl -s -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' https://api.example.com/healthz -v 2>&1 | grep -i traceparent

# Which services are actually reporting
curl -s localhost:8888/metrics | grep otelcol_receiver_accepted_spans

# Watch a live pipeline sample
kubectl logs -n observability deploy/otel-collector -f | grep -m5 'InstrumentationScope'

# Resource attributes an SDK will send
env | grep ^OTEL_

# Collector memory pressure
curl -s localhost:8888/metrics | grep -E 'otelcol_process_(memory_rss|runtime_heap_alloc_bytes)'