System design

Numbers to reason with #

Anything crossing a region costs at least the speed of light; no amount of tuning removes it. Design for one round trip per user action, not five.

Latency, throughput, saturation #

Throughput is work per unit time; latency is time per unit work; they trade against each other through queueing. As utilisation approaches 100%, queueing delay rises without bound — this is why a system at 80% CPU feels fine and the same system at 95% feels broken.

Measure percentiles, never means. An average of 100 ms with a p99 of 4 s means one request in a hundred is unacceptable, and every page composed of 50 calls hits it.

Scaling #

Scale the stateless tier first; it is the easy half. Almost every real limit is the database, and the fix is usually removing work rather than adding replicas.

Shard on a key with even distribution and no cross-shard queries in the hot path: user ID or tenant ID, rarely time. Time-based sharding puts every current write on one shard.

Caching #

key = "user:v2:7"       # version in the key: deploy invalidates without flushing
ttl = 300 + rand(0, 60) # jitter, or every key expires in the same second

Three defences worth building in from the start: jittered TTLs, a single-flight lock so one miss triggers one load, and a negative cache for “not found” so a missing key cannot be used to hammer the database.

Invalidation is the hard part. Prefer short TTLs and versioned keys over event-driven invalidation, which is correct in theory and wrong in the one path nobody updated.

Load balancing #

Health checks must exercise the dependency path that matters without cascading: a readiness check that queries the database takes every instance out at once when the database blips. Check liveness shallowly and readiness slightly deeper, and shed load rather than failing health checks under pressure.

Resilience #

Timeouts must shrink as you go deeper: if the edge allows 5 s, the service should allow 3 s and the database 1 s. Equal timeouts at every layer mean the whole chain waits for the slowest thing before anybody gives up.

Exponential backoff without jitter synchronises every client into a retry wave. sleep(random(0, min(cap, base * 2^attempt))) is the version that actually works.

Data #

One store of record; everything else is a derived projection that can be rebuilt. The moment two systems both claim to be authoritative, reconciliation becomes a permanent tax.

Replication is asynchronous unless you paid for it not to be: a read straight after a write can miss it. Route read-after-write to the primary, or carry a version and wait for it.

CAP in practice: during a partition you choose between serving stale data and refusing requests. Decide per endpoint — a product page can be stale, a payment cannot.

APIs #

GET    /v1/orders?status=open&limit=50&cursor=eyJ...    200
POST   /v1/orders                                        201 + Location
GET    /v1/orders/{id}                                   200 | 404
PATCH  /v1/orders/{id}                                   200 | 409
DELETE /v1/orders/{id}                                   204

Breaking changes need a new version; additive changes do not. A field that becomes required, an enum that gains a value clients must handle, or a default that changes are all breaking even when the schema still validates.

Asynchronous work #

Queues turn a synchronous dependency into a durable one, at the cost of eventual consistency and a new failure surface.

Multi-tenancy #

Whatever the model, enforce the tenant boundary in one place — a row-level policy or a repository layer — never by remembering to add WHERE tenant_id = ? in each query.

Noisy neighbours are the default failure: per-tenant quotas and rate limits are not optional once more than one tenant matters.

Reviewing a design #

Ask these in order, and stop when an answer is missing:

1. What is the request rate, the data volume and the growth rate?
2. Which parts must be strongly consistent, and which may be stale?
3. What happens when each dependency is slow, then when it is down?
4. Where is the state, and how is it restored after loss?
5. What is the blast radius of one bad deploy, one bad tenant, one bad key?
6. How would an operator detect this failing, and what would they do?
7. What is the rollback, and has anyone run it?