Networking
HTTP and curl
curl flags for diagnosis, timing breakdowns, TLS checks, proxy behaviour and what each status code actually tells you.
Cheatsheet #
| Task | Command |
|---|---|
| Headers only | curl -I https://example.com |
| Follow redirects | curl -L https://example.com |
| Fail the script on 4xx/5xx | curl -fsS https://example.com |
| Show the request too | curl -v https://example.com |
| Status code only | curl -o /dev/null -sw '%{http_code}\n' URL |
| Timing breakdown | curl -o /dev/null -sw '@curl-format.txt' URL |
| POST JSON | curl -X POST -H 'content-type: application/json' -d @body.json URL |
| Upload a file | curl -F file=@report.csv URL |
| Bearer token | curl -H "authorization: Bearer $TOKEN" URL |
| Pin to an IP, keep the host | curl --resolve api.example.com:443:10.0.0.5 https://api.example.com |
| Ignore certificate errors | curl -k URL (diagnosis only) |
| Force HTTP/1.1 or /2 | curl --http1.1 URL, curl --http2 URL |
| Through a proxy | curl -x http://proxy:3128 URL |
| Retry with backoff | curl --retry 3 --retry-delay 2 --retry-connrefused URL |
| Cap the time | curl --connect-timeout 3 --max-time 10 URL |
| Save cookies | curl -c jar -b jar URL |
A request that is failing #
curl -sSv https://api.example.com/health 2>&1 | grep -vE '^\{|^\}' # handshake, headers, status
curl -o /dev/null -sw 'code=%{http_code} dns=%{time_namelookup} conn=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' https://api.example.com/health
curl --resolve api.example.com:443:10.0.0.5 -sv https://api.example.com/health # bypass DNS, keep SNI and Host| Failure | Where it happened |
|---|---|
Could not resolve host | DNS — see DNS |
Connection refused | Reached the host, nothing listening on that port |
Connection timed out | Packet dropped: firewall, security group, wrong subnet |
SSL certificate problem | Chain, name mismatch, expiry, or missing CA in this trust store |
Long time_appconnect | TLS handshake cost — often OCSP or a large chain |
Long time_starttransfer, short everything else | The server is slow to generate the response |
Works with -k | Certificate validation, not connectivity |
Flags worth knowing #
| Flag | Effect |
|---|---|
-f | Non-zero exit on HTTP errors; essential in scripts |
-s / -S | Silent, but still print errors (-sS together) |
-L | Follow redirects; --max-redirs bounds it |
-i | Include response headers in the output |
-D - | Dump headers to stdout while the body goes elsewhere |
--compressed | Request and transparently decode gzip/br |
-A, -e | Set user agent and referer |
--data-binary | Send the payload exactly, no newline stripping |
-G | Turn -d fields into a query string |
--url-query | Append a URL-encoded query parameter |
-w | Write-out template: timings, sizes, redirect URL, status |
-Z | Parallel transfers for multiple URLs |
--trace-ascii - | Full wire dump when headers are not enough |
Sending data #
curl -X POST https://api.example.com/items \
-H 'content-type: application/json' \
-d '{"name":"widget","qty":4}'
curl -X POST https://api.example.com/items -d @payload.json -H 'content-type: application/json'
curl -F 'file=@report.csv' -F 'note=monthly' https://api.example.com/upload # multipart
curl -G https://api.example.com/search --data-urlencode 'q=name with spaces' --data-urlencode 'limit=10'
curl -X PATCH -H 'content-type: application/merge-patch+json' -d '{"qty":5}' https://api.example.com/items/1
curl -T ./artifact.tgz https://uploads.example.com/path/ # PUT a file-d implies POST with application/x-www-form-urlencoded, so JSON needs the header set explicitly. -d @file strips newlines; --data-binary @file does not, which matters for signatures and for YAML payloads.
Timing #
cat > curl-format.txt <<'EOF'
dns %{time_namelookup}s
connect %{time_connect}s
tls %{time_appconnect}s
waiting %{time_starttransfer}s
total %{time_total}s
size %{size_download} bytes
code %{http_code}
EOF
curl -o /dev/null -s -w '@curl-format.txt' https://api.example.com/healthSubtract to isolate: time_connect - time_namelookup is the TCP handshake, time_appconnect - time_connect is TLS, time_starttransfer - time_appconnect is the server thinking.
# Ten samples, to separate a slow server from a slow tail
for i in $(seq 10); do curl -o /dev/null -sw '%{time_total}\n' https://api.example.com/health; done | sort -nTLS #
curl -vI https://example.com 2>&1 | grep -E 'SSL connection|subject|issuer|expire'
curl --tlsv1.3 --tls-max 1.3 -sI https://example.com # force a version
curl --cacert /path/ca.pem https://internal.example.com
curl --cert client.pem --key client.key https://mtls.example.com
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | openssl x509 -noout -dates -subject -issuer-k proves the problem is validation and nothing more. Never leave it in a script: it disables the guarantee the certificate exists to provide. See TLS for chain repair.
Proxies #
curl -x http://proxy.example.com:3128 https://api.example.com
export http_proxy=http://proxy:3128 https_proxy=http://proxy:3128 no_proxy=.internal,localhost,10.0.0.0/8
curl --proxy-user user:pass -x http://proxy:3128 URL
curl -x socks5h://localhost:1080 URL # h = resolve DNS at the proxy
curl -sv https://api.example.com 2>&1 | grep -i 'CONNECT\|proxy'no_proxy matching is prefix-based on hostname, so 10.0.0.0/8 works with some clients and not others — test rather than assume. A proxy in the path shows up as a CONNECT in verbose output and often as extra via or x-cache headers.
Status codes in practice #
| Code | What it usually means here |
|---|---|
301 / 308 | Permanent redirect; 308 preserves the method and body |
302 / 307 | Temporary; 307 preserves the method |
400 | Malformed request — check content type and body encoding |
401 | No or invalid credentials; look for www-authenticate |
403 | Authenticated but not permitted, or blocked by a WAF |
404 | Wrong path, or the route exists only on another host/vhost |
405 | Right path, wrong method |
409 | Conflict: optimistic concurrency, duplicate create |
413 / 414 | Body or URL too large for the server or proxy |
429 | Rate limited; honour retry-after |
499 | nginx-specific: client closed before the response |
500 | The application threw; its logs have the answer |
502 | Proxy could not reach the backend, or got a malformed reply |
503 | Backend unavailable: no healthy upstreams, or load shedding |
504 | Backend accepted but did not answer within the proxy’s timeout |
A 502 and a 504 from the same proxy point at different problems: 502 is connection-level, 504 is a timeout. Which timeout is in play is usually visible in the proxy’s own configuration, not the application’s.
In scripts #
if ! body=$(curl -fsS --max-time 10 --retry 3 --retry-connrefused https://api.example.com/health); then
printf 'health check failed\n' >&2
exit 1
fi
code=$(curl -o /tmp/body -D /tmp/headers -sw '%{http_code}' -X POST -d @payload.json -H 'content-type: application/json' https://api.example.com/items)
case $code in
20*) ;;
429) sleep "$(awk 'tolower($1) ~ /^retry-after:/ {print $2}' /tmp/headers | tr -d '\r')" ;;
*) printf 'unexpected %s: %s\n' "$code" "$(cat /tmp/body)" >&2; exit 1 ;;
esacAlways set --max-time: without it a hung server hangs the job, the pipeline and eventually the runner.
Oneliners #
# Status code only
curl -o /dev/null -sw '%{http_code}\n' https://example.com
# Follow redirects and show each hop
curl -sIL https://example.com | awk '/^HTTP|^location/'
# Final URL after redirects
curl -o /dev/null -sw '%{url_effective}\n' -L https://example.com
# Response headers only, sorted
curl -sD - -o /dev/null https://example.com | sort
# Poll until a service is healthy
until curl -fsS --max-time 2 http://localhost:8080/health >/dev/null; do sleep 1; done
# Hammer an endpoint 50 times in parallel and summarise codes
seq 50 | xargs -P10 -I{} curl -o /dev/null -sw '%{http_code}\n' https://api.example.com/ | sort | uniq -c
# Compare two environments' responses
diff <(curl -s https://staging.example.com/api/config | jq -S .) <(curl -s https://prod.example.com/api/config | jq -S .)
# Download with resume
curl -C - -O https://example.com/big.iso
# Send a signed webhook payload
body='{"event":"test"}'; sig=$(printf '%s' "$body" | openssl dgst -sha256 -hmac "$SECRET" -hex | awk '{print $2}'); curl -X POST -H "x-signature: sha256=$sig" -d "$body" https://api.example.com/hook
# Check which HTTP version and ALPN was negotiated
curl -so /dev/null -w '%{http_version} %{scheme}\n' https://example.com
# Request with a fixed Host header against an IP (vhost testing)
curl -H 'Host: api.example.com' http://10.0.0.5/health
# Measure the slowest of N endpoints
for u in "${urls[@]}"; do printf '%s %s\n' "$(curl -o /dev/null -sw '%{time_total}' "$u")" "$u"; done | sort -rn | head