Software Engineering Wiki

Networking

HTTP and curl

curl flags for diagnosis, timing breakdowns, TLS checks, proxy behaviour and what each status code actually tells you.

Cheatsheet #

TaskCommand
Headers onlycurl -I https://example.com
Follow redirectscurl -L https://example.com
Fail the script on 4xx/5xxcurl -fsS https://example.com
Show the request toocurl -v https://example.com
Status code onlycurl -o /dev/null -sw '%{http_code}\n' URL
Timing breakdowncurl -o /dev/null -sw '@curl-format.txt' URL
POST JSONcurl -X POST -H 'content-type: application/json' -d @body.json URL
Upload a filecurl -F file=@report.csv URL
Bearer tokencurl -H "authorization: Bearer $TOKEN" URL
Pin to an IP, keep the hostcurl --resolve api.example.com:443:10.0.0.5 https://api.example.com
Ignore certificate errorscurl -k URL (diagnosis only)
Force HTTP/1.1 or /2curl --http1.1 URL, curl --http2 URL
Through a proxycurl -x http://proxy:3128 URL
Retry with backoffcurl --retry 3 --retry-delay 2 --retry-connrefused URL
Cap the timecurl --connect-timeout 3 --max-time 10 URL
Save cookiescurl -c jar -b jar URL

A request that is failing #

curl -sSv https://api.example.com/health 2>&1 | grep -vE '^\{|^\}'   # handshake, headers, status
curl -o /dev/null -sw 'code=%{http_code} dns=%{time_namelookup} conn=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' https://api.example.com/health
curl --resolve api.example.com:443:10.0.0.5 -sv https://api.example.com/health   # bypass DNS, keep SNI and Host
FailureWhere it happened
Could not resolve hostDNS — see DNS
Connection refusedReached the host, nothing listening on that port
Connection timed outPacket dropped: firewall, security group, wrong subnet
SSL certificate problemChain, name mismatch, expiry, or missing CA in this trust store
Long time_appconnectTLS handshake cost — often OCSP or a large chain
Long time_starttransfer, short everything elseThe server is slow to generate the response
Works with -kCertificate validation, not connectivity

Flags worth knowing #

FlagEffect
-fNon-zero exit on HTTP errors; essential in scripts
-s / -SSilent, but still print errors (-sS together)
-LFollow redirects; --max-redirs bounds it
-iInclude response headers in the output
-D -Dump headers to stdout while the body goes elsewhere
--compressedRequest and transparently decode gzip/br
-A, -eSet user agent and referer
--data-binarySend the payload exactly, no newline stripping
-GTurn -d fields into a query string
--url-queryAppend a URL-encoded query parameter
-wWrite-out template: timings, sizes, redirect URL, status
-ZParallel transfers for multiple URLs
--trace-ascii -Full wire dump when headers are not enough

Sending data #

curl -X POST https://api.example.com/items \
  -H 'content-type: application/json' \
  -d '{"name":"widget","qty":4}'

curl -X POST https://api.example.com/items -d @payload.json -H 'content-type: application/json'
curl -F 'file=@report.csv' -F 'note=monthly' https://api.example.com/upload   # multipart
curl -G https://api.example.com/search --data-urlencode 'q=name with spaces' --data-urlencode 'limit=10'
curl -X PATCH -H 'content-type: application/merge-patch+json' -d '{"qty":5}' https://api.example.com/items/1
curl -T ./artifact.tgz https://uploads.example.com/path/    # PUT a file

-d implies POST with application/x-www-form-urlencoded, so JSON needs the header set explicitly. -d @file strips newlines; --data-binary @file does not, which matters for signatures and for YAML payloads.

Timing #

cat > curl-format.txt <<'EOF'
     dns  %{time_namelookup}s
 connect  %{time_connect}s
     tls  %{time_appconnect}s
 waiting  %{time_starttransfer}s
   total  %{time_total}s
    size  %{size_download} bytes
    code  %{http_code}
EOF

curl -o /dev/null -s -w '@curl-format.txt' https://api.example.com/health

Subtract to isolate: time_connect - time_namelookup is the TCP handshake, time_appconnect - time_connect is TLS, time_starttransfer - time_appconnect is the server thinking.

# Ten samples, to separate a slow server from a slow tail
for i in $(seq 10); do curl -o /dev/null -sw '%{time_total}\n' https://api.example.com/health; done | sort -n

TLS #

curl -vI https://example.com 2>&1 | grep -E 'SSL connection|subject|issuer|expire'
curl --tlsv1.3 --tls-max 1.3 -sI https://example.com    # force a version
curl --cacert /path/ca.pem https://internal.example.com
curl --cert client.pem --key client.key https://mtls.example.com
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | openssl x509 -noout -dates -subject -issuer

-k proves the problem is validation and nothing more. Never leave it in a script: it disables the guarantee the certificate exists to provide. See TLS for chain repair.

Proxies #

curl -x http://proxy.example.com:3128 https://api.example.com
export http_proxy=http://proxy:3128 https_proxy=http://proxy:3128 no_proxy=.internal,localhost,10.0.0.0/8
curl --proxy-user user:pass -x http://proxy:3128 URL
curl -x socks5h://localhost:1080 URL          # h = resolve DNS at the proxy
curl -sv https://api.example.com 2>&1 | grep -i 'CONNECT\|proxy'

no_proxy matching is prefix-based on hostname, so 10.0.0.0/8 works with some clients and not others — test rather than assume. A proxy in the path shows up as a CONNECT in verbose output and often as extra via or x-cache headers.

Status codes in practice #

CodeWhat it usually means here
301 / 308Permanent redirect; 308 preserves the method and body
302 / 307Temporary; 307 preserves the method
400Malformed request — check content type and body encoding
401No or invalid credentials; look for www-authenticate
403Authenticated but not permitted, or blocked by a WAF
404Wrong path, or the route exists only on another host/vhost
405Right path, wrong method
409Conflict: optimistic concurrency, duplicate create
413 / 414Body or URL too large for the server or proxy
429Rate limited; honour retry-after
499nginx-specific: client closed before the response
500The application threw; its logs have the answer
502Proxy could not reach the backend, or got a malformed reply
503Backend unavailable: no healthy upstreams, or load shedding
504Backend accepted but did not answer within the proxy’s timeout

A 502 and a 504 from the same proxy point at different problems: 502 is connection-level, 504 is a timeout. Which timeout is in play is usually visible in the proxy’s own configuration, not the application’s.

In scripts #

if ! body=$(curl -fsS --max-time 10 --retry 3 --retry-connrefused https://api.example.com/health); then
  printf 'health check failed\n' >&2
  exit 1
fi

code=$(curl -o /tmp/body -D /tmp/headers -sw '%{http_code}' -X POST -d @payload.json -H 'content-type: application/json' https://api.example.com/items)
case $code in
  20*) ;;
  429) sleep "$(awk 'tolower($1) ~ /^retry-after:/ {print $2}' /tmp/headers | tr -d '\r')" ;;
  *)   printf 'unexpected %s: %s\n' "$code" "$(cat /tmp/body)" >&2; exit 1 ;;
esac

Always set --max-time: without it a hung server hangs the job, the pipeline and eventually the runner.

Oneliners #

# Status code only
curl -o /dev/null -sw '%{http_code}\n' https://example.com

# Follow redirects and show each hop
curl -sIL https://example.com | awk '/^HTTP|^location/'

# Final URL after redirects
curl -o /dev/null -sw '%{url_effective}\n' -L https://example.com

# Response headers only, sorted
curl -sD - -o /dev/null https://example.com | sort

# Poll until a service is healthy
until curl -fsS --max-time 2 http://localhost:8080/health >/dev/null; do sleep 1; done

# Hammer an endpoint 50 times in parallel and summarise codes
seq 50 | xargs -P10 -I{} curl -o /dev/null -sw '%{http_code}\n' https://api.example.com/ | sort | uniq -c

# Compare two environments' responses
diff <(curl -s https://staging.example.com/api/config | jq -S .) <(curl -s https://prod.example.com/api/config | jq -S .)

# Download with resume
curl -C - -O https://example.com/big.iso

# Send a signed webhook payload
body='{"event":"test"}'; sig=$(printf '%s' "$body" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}'); curl -X POST -H "x-signature: sha256=$sig" -d "$body" https://api.example.com/hook

# Check which HTTP version and ALPN was negotiated
curl -so /dev/null -w '%{http_version} %{scheme}\n' https://example.com

# Request with a fixed Host header against an IP (vhost testing)
curl -H 'Host: api.example.com' http://10.0.0.5/health

# Measure the slowest of N endpoints
for u in "${urls[@]}"; do printf '%s %s\n' "$(curl -o /dev/null -sw '%{time_total}' "$u")" "$u"; done | sort -rn | head

Last updated 15 September 2026 · Edit this page