System Design Interview Prep

Curated system design interview questions, foundational concepts, distributed-system patterns, and end-to-end design problems.

Topics covered:

• Fundamentals: CAP, PACELC, ACID vs BASE, consistency models, latency vs throughput, back-of-envelope estimation
• Scalability: vertical vs horizontal, stateless services, load balancing, CDN
• Data: SQL vs NoSQL, indexing, sharding, replication, partitioning, secondary indexes
• Caching: layers, eviction, write strategies, invalidation, cache stampede
• Messaging: queues, pub/sub, brokers (Kafka, Rabbit, SQS), exactly-once
• Communication: REST, gRPC, GraphQL, WebSockets, Server-Sent Events, long-polling
• Distributed systems: consensus (Raft, Paxos), clocks (Lamport, vector), gossip, leader election
• Reliability: redundancy, retries, backoff, circuit breakers, bulkheads, idempotency
• Patterns: CQRS, event sourcing, saga, outbox, sidecar, strangler fig, BFF
• Security: AuthN/AuthZ, OAuth2/OIDC, JWT, rate limiting, OWASP
• Observability: metrics, traces, logs, SLO/SLI/SLA, error budgets
• Top design problems: URL shortener, news feed, chat, ride-share, rate limiter, distributed cache, search, notifications, video streaming, payments, ad-click pipeline, recommendation, file storage, web crawler

Q: What are CAP and PACELC, and how do they shape distributed-system design?

Answer:

CAP is a forced-choice theorem about distributed databases under network failure. PACELC extends it to the non-failure case, where latency vs consistency is the real trade. Together they describe almost every storage system on the market.

CAP Theorem (Brewer, 2000)

In the presence of a network partition, a system can provide at most two of:

• C — Consistency: every read sees the most recent committed write (linearizability).
• A — Availability: every request gets a non-error response.
• P — Partition tolerance: system keeps working despite dropped/delayed messages.

Critical: P is not optional in any real distributed system. Networks fail. So the actual choice is CP vs AP during a partition.

                     Partition occurs
                          │
            ┌─────────────┴──────────────┐
            ▼                            ▼
        CP system                    AP system
   refuse some requests          accept all requests
   (return errors)               (may return stale data)
   guarantees consistency        guarantees availability

Example Classifications

PACELC (Abadi, 2012)

CAP only describes partition behavior. PACELC adds the steady-state question:

If Partition: choose A or C. Else (no partition): choose L (Latency) or C (Consistency).

A system is described by both halves: PA/EL (DynamoDB), PC/EC (Spanner), PA/EC (MongoDB sometimes), PC/EL (some configs).

The "EL or EC" axis matters because synchronous replication to remote replicas adds latency. Choosing EC means waiting for ack from N regions before responding to the client.

Tunable Consistency

Modern systems rarely pick one corner. They expose per-operation knobs:

-- Postgres: synchronous_commit per transaction
SET LOCAL synchronous_commit = on;        -- wait for replica fsync

-- Cassandra
SELECT ... USING CONSISTENCY QUORUM;       -- per query

-- DynamoDB
GetItem(ConsistentRead=true)               -- per operation

You buy consistency only where you need it. Eventual reads for product browse; strong reads for checkout.

Quorum Math

For N replicas:

W + R > N   →  strong consistency on reads
W + R ≤ N   →  eventually consistent

Common configs:

• N=3, W=2, R=2: tolerates 1 node loss, strong reads via quorum.
• N=3, W=3, R=1: writes block on full ISR; reads are fast.
• N=3, W=1, R=1: AP — fast, eventually consistent.

What CAP Is NOT

• Not about whether one node is down. Single-node failures are a different problem.
• Not a static label for an entire system. Same DB can run CP or AP depending on config.
• Not a green light to pick "any two." You always need P.

Interview Use

When asked "what's the consistency model of your design?":

1. State the failure mode you're optimizing for.
2. Choose CP or AP per data class, not for the whole system. Orders → CP. Likes → AP.
3. Mention quorum settings, replication topology, and what happens during partition (errors vs stale reads).

Common Mistakes

[!NOTE] CAP is the partition-time trade-off. PACELC is the steady-state trade-off. Real systems live mostly in steady state — PACELC matters more day-to-day, but CAP determines what happens when the network actually breaks.

Interview Follow-ups

• "How would you build a CP system on top of an AP store?" — Wrap with a consensus layer (Raft) above the keyspace; trade availability for linearizability.
• "Why doesn't Spanner break CAP?" — It doesn't. It's CP. TrueTime narrows the wait window to ~7 ms, making the consistency cost cheap.
• "What's linearizability vs serializability?" — Linearizability is about single-object real-time ordering. Serializability is about transactions being equivalent to some serial order. SQL DBs offer the latter; distributed KVs argue about the former.

Q: ACID vs BASE — what guarantees, and when to pick which?

Answer:

ACID is the transactional guarantee of relational databases: writes are all-or-nothing, isolated, and durable. BASE is the loose guarantee of NoSQL and distributed systems: writes eventually settle. Picking between them per-data-class is one of the most consequential design decisions in any system.

ACID

• Atomicity: a transaction succeeds entirely or fails entirely. No partial commits.
• Consistency: a transaction leaves the database in a valid state (constraints, FKs, triggers).
• Isolation: concurrent transactions don't see each other's intermediate state.
• Durability: once committed, the data survives crashes (fsync to disk or replicated WAL).

Example: bank transfer.

BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;

Either both rows change, or neither. No middle state where money disappears.

Isolation Levels

The "I" in ACID is not binary — SQL standard defines four levels:

Anomalies:

• Dirty read: read uncommitted data from another tx.
• Non-repeatable read: same query, two values in one tx.
• Phantom read: same range query, different row count.
• Lost update: two tx read, both write, second overwrites first.
• Write skew: two tx read-then-write disjoint rows under a constraint that should hold globally.

SERIALIZABLE prevents all but is the slowest. Most apps run READ_COMMITTED and add SELECT ... FOR UPDATE or version columns to guard hot rows.

BASE

• Basically Available: system responds (maybe with stale data).
• Soft state: state may change over time without input (replicas converge).
• Eventual consistency: given no new writes, all replicas converge.

Trade: relaxed consistency in exchange for:

• Higher availability under partitions.
• Lower latency (no synchronous cross-region writes).
• Horizontal scale beyond what 2PC allows.

Example: a social-network "like count." Showing 1,234 when it's truly 1,235 is fine.

How to Decide

Per data class, ask:

In one system you almost always have both:

• ACID for orders, payments, inventory, user identity.
• BASE for feeds, recommendations, analytics counters, search indexes.

Mixing Them Safely

Common pattern: write to ACID system of record, asynchronously propagate to BASE views.

Postgres (orders, ACID)
   │
   ▼ CDC (Debezium / outbox)
   ▼
Kafka
   │
   ▼
   ├──► ElasticSearch (search index)
   ├──► Redis (cache)
   └──► ClickHouse (analytics)

Source of truth is ACID. Derived views are BASE. Reconcile drift via periodic full-sync jobs or bookkeeping.

Quasi-ACID at Scale

• Spanner / CockroachDB: distributed ACID. External consistency via TrueTime / hybrid logical clocks. Slower than single-node PG, faster than 2PC.
• DynamoDB Transactions: ACID across up to 100 items. Adds latency.
• MongoDB multi-document transactions: ACID within a shard; cross-shard adds 2PC.

These narrow the gap but don't eliminate it. Cost (latency, throughput) is real.

Common Mistakes

[!NOTE] ACID is a feature you pay for. BASE is a property you build around. The biggest engineering wins come from segmenting your data correctly: ACID where wrongness costs money, BASE everywhere else.

Interview Follow-ups

• "How would you maintain a global like-count without write contention?" — Sharded counter; per-shard atomic increment; periodic aggregation. BASE; final number eventually consistent.
• "How do you keep an ACID DB and a search index in sync?" — Outbox table in the same transaction; CDC stream → indexer. Avoid dual writes.
• "What does SELECT FOR UPDATE SKIP LOCKED solve?" — Lets multiple workers pull from a job table without blocking each other or pulling the same row.

Q: Explain consistency models — strong, eventual, causal, read-your-writes, monotonic.

Answer:

"Consistency" is overloaded — it means one thing in CAP, another in ACID, and a third in distributed-systems literature. The literature definition is the one that matters at design time: what guarantees does a read make about the writes it observes?

The Spectrum

Strongest                                                           Weakest
─────────────────────────────────────────────────────────────────────────►

Strict Serializable   Linearizable   Sequential   Causal   Eventual

Roughly: the further left, the more coordination required and the slower the system.

Strong Consistency (Linearizability)

A read returns the most recent committed write in real time. As if there were one copy of the data and operations happened one at a time.

T1: write(x, 1)
T2: write(x, 2)         (after T1 commits)
T3: read(x)             must return 2

Achieved via consensus (Raft/Paxos), single-leader writes with synchronous replication, or external clocks (Spanner).

Cost: latency = round-trip to quorum. Cannot be globally faster than the speed of light × distance.

Sequential Consistency

Operations appear in some total order consistent with each process's program order, but not necessarily the real-time order.

Weaker than linearizable. Strong enough for many algorithms (caches, mutexes).

Causal Consistency

Operations that are causally related are seen in the same order by all replicas. Concurrent operations can be observed in any order.

T1: post(p1)
T2: comment(p1, c1)         depends on p1
                            → every reader sees p1 before c1
T3: post(p2)                concurrent with above
                            → readers may see p2 before or after p1/c1

Implemented with vector clocks. Practical for social feeds, collaborative editing.

Read-Your-Writes

A user always sees their own writes. Other users may not see them yet.

You post a photo.
You immediately refresh — you see the photo.
Your follower refreshes — may not see it yet.

Common pattern: route a user's reads to the same replica (or leader) as their writes for a short window. Or buffer the write client-side until visible.

Monotonic Reads

If a process reads value v1, subsequent reads will see v1 or newer. Never goes backward.

Without monotonic reads, a load-balanced read might bounce between replicas with different lag and the user sees state regress. Stick a user to a replica (sticky sessions) to fix.

Monotonic Writes

A process's writes are applied in the order it issued them. Often paired with read-your-writes.

Eventual Consistency

Given no new writes, all replicas converge to the same value. No bound on when.

Strongest claim: "eventually" — useless for SLAs unless you add bounded staleness.

Cassandra Read One:    nearest replica wins, may be stale by seconds
DynamoDB eventual:     up to ~1s lag typical
S3:                    strong consistency since 2020

Bounded Staleness

A practical strengthening of eventual: read no more than T seconds (or N operations) behind the latest committed write.

Cosmos DB exposes this as a knob: "5 seconds" or "100 writes" max lag. Useful for dashboards that can be slightly behind but not arbitrarily so.

Choosing Per Data Class

Where Each Lives

Read-Your-Writes Implementations

1. Sticky session. Send each user to the leader (or same replica) for a window after a write.

2. Version pinning. Client stores the write's version; reads include min_version, server waits or routes to a replica that has it.

3. Client-side cache. App holds the just-written value locally until the server's view catches up.

Causal Consistency Implementations

1. Vector clocks. Each replica tags writes with a vector of per-node counters. Reader walks dependencies.

2. Lamport clocks + dependency tracking. Simpler but less precise.

3. CRDT (conflict-free replicated data types). Some CRDTs are causal by construction.

Common Mistakes

[!NOTE] In interviews, when asked "what consistency does this need?", the wrong answer is "strong" by default. The right answer is "linearizable for X, causal for Y, eventual for Z, with bounded staleness of ~T."

Interview Follow-ups

• "What's the difference between linearizable and serializable?" — Linearizable: real-time order on a single object. Serializable: transactions equivalent to some serial schedule. Strict serializable = both.
• "How does Spanner achieve external consistency?" — TrueTime (GPS + atomic clocks bound clock skew); commits wait out the uncertainty window.
• "Why is eventual consistency cheap?" — No coordination on the write path; replicas reconcile in the background.

Q: Latency vs throughput vs tail latency — why does p99 matter more than average?

Answer:

Two systems can have the same average latency and feel completely different to users. The difference is the distribution. Modern system design is dominated by the long tail (p99, p99.9), not by the mean.

Definitions

• Latency: time for one request, end to end.
• Throughput: requests served per unit time.
• Bandwidth: bytes per unit time.

They aren't independent: by Little's Law, concurrent_requests = throughput × latency. If you fix concurrency, raising one drops the other.

Why Averages Lie

Request latencies (ms):
  20, 25, 22, 30, 27, 25, 28, 1200, 22, 26
                                ^
  Average: 142 ms                slow tail
  Median:  26 ms
  p99:    1200 ms

Average gets dragged by a single outlier. Median + percentiles tell the real story.

Percentiles Cheat Sheet

Fan-Out Multiplies Tail Latency

A page that issues N parallel calls and waits for all returns when the slowest one returns. If each call has 1% chance of being slow:

N = 1:    1% of requests slow
N = 10:   1 - 0.99^10  ≈ 10% slow
N = 100:  1 - 0.99^100 ≈ 63% slow!

p99 of the service becomes p50 of the page. This is why backend latency p99 matters more than average — fan-out amplifies it.

Sources of Tail Latency

• GC pauses (long for big heaps, short with ZGC).
• JIT warmup on cold instances.
• Cache misses to slower storage.
• Lock contention under load.
• Network jitter / packet loss → retransmit.
• Disk fsync / log rotation.
• Cold connections (TCP slow start, TLS handshake).
• Coordinated omission: load generator stops counting when it can't push, hiding the real tail.

Latency Numbers to Remember

DC round-trip:           ~500 µs
Cross-AZ:                ~1–2 ms
Cross-region (US ↔ EU):  ~80–100 ms
Cross-continent:         ~150 ms
Memory read:             ~100 ns
SSD random read:         ~100 µs
HDD seek:                ~10 ms

A simple "fetch from DB" query in another region is automatically ~100 ms before you do any work.

Throughput Tuning

Increase throughput via:

• Parallelism (multiple threads/cores).
• Pipelining (overlap requests; multiple in flight per connection).
• Batching (Kafka producer linger, DB multi-row insert).
• Async I/O (don't block threads on remote calls).
• Avoid contention (locks, hot rows, GC).

Each often hurts latency for individual requests in exchange for system-wide throughput.

Latency Tuning

Reduce latency via:

• Cache (RAM-fast vs disk/network).
• Move compute to data (e.g., DB-side aggregation).
• Reduce serial dependencies (parallelize independent calls).
• Reduce hops (CDN, edge compute).
• Connection reuse (HTTP keep-alive, gRPC streams).
• Pre-warm caches and connection pools.

Throughput vs Latency Trade-off

Throughput
   │             ╱╲   ← optimal point
   │           ╱    ╲
   │         ╱        ╲      ← degradation
   │       ╱            ╲
   │     ╱
   │   ╱
   │ ╱
   └──────────────────────────►   Concurrency / load

Past the optimal point, adding load reduces throughput because queues fill, contention rises, GC churns. The same shape governs every queueing system.

Load Shedding > Queuing

When a service is overloaded, queueing more requests makes it worse — each new request sits behind a longer queue, increasing latency, until upstream times out and retries. Better: shed requests at the door.

Healthy:    accept all requests
Degraded:   accept critical, reject low-priority
Overloaded: reject all but health checks, surface clear 503

Hedged Requests

Tail latency mitigation: send a duplicate request to a second replica after a short delay (p95). Take whichever returns first. Cancels the slow one if possible.

Pioneered by Google for search. Cheap insurance against a single slow node.

t=0:    send to A
t=p95:  if no response, send to B
t=any:  return first response, cancel the other

Coordinated Omission

A common benchmark bug: load generator can't keep up with target QPS, so it backs off — the slow responses it's measuring don't include the time spent waiting to send. Real-world load doesn't get to back off.

Use wrk2, tlb-bench, or modern tools that compensate. Otherwise your published p99 is fiction.

Latency Budgets

In an SLA of 200 ms p99 end to end:

Browser → CDN:       30 ms     (network)
CDN → API gateway:    5 ms
Gateway → service:    1 ms
Service compute:     50 ms
Service → cache:      2 ms
Cache miss → DB:     20 ms
Serialization:        5 ms
Network back:        30 ms
                    ────
                    143 ms      (within budget)

Each layer must fit within its slice. Cache miss on every request blows the budget.

Common Mistakes

[!NOTE] The most operationally important number in any service is its p99 latency under realistic load. Track it. Alert on it. Tune it. Average latency is a vanity metric.

Interview Follow-ups

• "How do you measure latency under load?" — Distributed tracing (sample); aggregate via histogram metrics (Prometheus + histogram_quantile); load test with wrk2 or k6.
• "What's a hedged request, and when not to use it?" — Don't use for non-idempotent writes; don't use when load is already high (doubles work).
• "Why is p99 a worse SLO than latency ≤ X for 99% of requests / minute?" — Both describe the same thing if framed right; the second form is what alerting actually does — windowed.

Q: How do you do back-of-envelope estimation in a system design interview?

Answer:

Most system design interviews expect numeric reasoning: storage in TB, requests per second, bandwidth in Gbps. The numbers don't have to be perfect — they have to be defensible and order-of-magnitude correct. Walk through assumptions out loud.

Numbers Worth Memorizing

Time (orders of magnitude)
   L1 cache reference                           1 ns
   Branch mispredict                            5 ns
   L2 cache reference                           7 ns
   Mutex lock/unlock                           25 ns
   Main memory reference                      100 ns
   Compress 1KB with Zippy                  3,000 ns
   Send 2KB over 1 Gbps network            20,000 ns   = 20 µs
   SSD random read                        150,000 ns   = 150 µs
   Read 1MB sequentially from memory      250,000 ns   = 250 µs
   Round-trip within same datacenter      500,000 ns   = 500 µs
   Read 1MB sequentially from SSD       1,000,000 ns   = 1 ms
   Disk seek (HDD)                     10,000,000 ns   = 10 ms
   Read 1MB sequentially from HDD      20,000,000 ns   = 20 ms
   Send packet CA → Netherlands → CA  150,000,000 ns   = 150 ms

Source: Jeff Dean's classic table, updated for SSDs.

Capacity Cheat Sheet

Server RAM:           commodity 64–256 GB, large 1–2 TB
Server cores:         8–96
Server NIC:           10–25 Gbps typical, 100 Gbps possible
SSD throughput:       3–7 GB/s sequential read, 100k+ IOPS random
HDD throughput:       150 MB/s sequential, 100 IOPS random
Postgres on SSD:      ~5,000–50,000 simple QPS per node depending on row size
Redis single node:    100k–1M ops/sec
Kafka broker:         100–500 MB/s sustained
S3:                   ~3,500 PUTs/sec/prefix, ~5,500 GETs/sec/prefix

Quantities Worth Memorizing

Seconds in a day:     86,400 ≈ 10⁵
Seconds in a year:    31,536,000 ≈ 3 × 10⁷ (~ π × 10⁷)
1 KB:                 10³ bytes (or 2¹⁰; close enough)
1 MB:                 10⁶
1 GB:                 10⁹
1 TB:                 10¹²
1 PB:                 10¹⁵
1B users:             1,000,000,000 = 10⁹
1B/day:               ≈ 11,574 / sec
1B/year:              ≈ 32 / sec

The Workflow

When the interviewer says "design X for Y users":

1. Functional scope. Pick 2–4 core features. Don't try to design everything.

2. Scale assumptions. DAU (daily active users) → QPS → storage → bandwidth.

3. QPS. Reads vs writes. Peak ≈ 2–3 × average.

4. Storage. Bytes per record × records per day × retention.

5. Bandwidth. QPS × payload size.

Worked Example: Twitter Read Path

"Design Twitter for 200M DAU. 100 tweets/day read per user, 1 tweet/day written."

QPS:

Reads:  200M × 100 / 86,400 ≈ 230k QPS average
Writes: 200M × 1   / 86,400 ≈ 2.3k QPS average
Peak: 2–3× average → reads ~700k QPS, writes ~7k QPS

Storage growth:

Per tweet: 200 B (text) + 100 B (metadata) ≈ 300 B
Per day:   200M × 1 × 300 B = 60 GB
Per year:  60 GB × 365 ≈ 22 TB
5-year archive: ~110 TB (text only)
+ media (photo/video) — typically 10–100× larger

Bandwidth:

Read egress: 700k × 300 B ≈ 200 MB/s ≈ 1.6 Gbps
(plus media — much larger; usually served by CDN)

Conclusion: a few hundred servers for the read tier, fronted by CDN for media. One Postgres can't take 700k QPS — needs caching (Redis) and replication.

Worked Example: URL Shortener

"200M shortenings/day, 20B reads/day."

Write QPS:   200M / 86,400 ≈ 2,300/s
Read QPS:    20B  / 86,400 ≈ 230,000/s
Read/write ratio ≈ 100:1

Storage:

Per record:  ~500 B (long URL + short code + metadata + indexes)
Per day:     200M × 500 B = 100 GB
Per year:    100 GB × 365 ≈ 36 TB
5-year keep: ~180 TB

Decision: read tier dominated by cache. Origin DB can be smaller. Use Redis or in-memory LRU with TTL. Object storage for archive (rarely-accessed old links).

Common Conversions

1 Gbps   = 125 MB/s
10 Gbps  = 1.25 GB/s
100 Gbps = 12.5 GB/s

Latency budgets
  Web page first paint:           < 1 s
  REST API p99:                   < 200 ms (good), < 1 s (acceptable)
  Cache hit:                      < 10 ms
  DB query (indexed):             < 5 ms in DC
  Cross-region call:              50–150 ms

Heuristics

• Daily → per-second: divide by ~10⁵.
• Peak factor: 2–3× for global apps; 5–10× for spikes (Black Friday, viral content).
• Read/write ratio: most consumer apps are 10:1 to 1000:1 read-heavy.
• Hot data: 80/20 rule — 20% of keys account for 80% of accesses. Cache the 20%.
• Index overhead: ~30% on top of row size for typical Postgres tables.
• Replication factor: ×3 for storage cost in most replicated DBs.

Validating With Constraints

Sanity-check by mapping numbers to known systems:

• "700k QPS" → larger than any single Postgres node; need sharding or caching.
• "100 TB" → fits in a single dedicated cluster, no need for object storage tier.
• "10 PB" → object storage is mandatory; pricing matters more than performance.

Common Mistakes

[!NOTE] Interviewers care more about how you reason than the exact answer. State assumptions, do arithmetic out loud, and revise when given a constraint. "I assumed 200M DAU, 100 reads each — does that match?" is a great prompt.

Interview Follow-ups

• "How would you handle 10× the load?" — Identify the bottleneck from the numbers (writes, reads, storage, bandwidth) and propose specific scaling (sharding, caching, CDN).
• "How much does this cost?" — TB × $/TB for storage, QPS × $/req for managed services, Gbps × $/GB for egress.
• "What's the latency budget end-to-end?" — Decompose into client → CDN → LB → app → cache → DB; sum the worst-case to compare against SLA.

Q: What is scalability, and what are the axes you can scale along?

Answer:

A scalable system handles more load by adding resources, ideally proportionally (2× resources → 2× capacity). The three classical axes — vertical, horizontal, and functional — describe different strategies. Real systems scale along all three at once.

The Scale Cube

Coined by Martin Abbott, this is the canonical mental model:

                  Z-axis: data partition / shard
                       ▲
                       │
                       │
                       │
       Y-axis (functional split, services)
              ◄────────┼────────►
                       │
                       │
                       ▼
                  X-axis: clone / horizontal

• X: run more copies of the same thing behind a load balancer.
• Y: split by function — separate services for orders, payments, search.
• Z: split by data — shard customers by region, user_id mod N.

X-Axis: Horizontal Scaling

Run N identical app servers behind a load balancer. Prerequisites:

• Stateless application tier (no in-memory session that survives across requests).
• Externalize state to a shared store (Redis, DB).
• Idempotent request handling (because retries may hit different instances).

        ┌──── App-1
LB ─────┼──── App-2     → DB / cache / queue
        └──── App-3

Cheap if you've designed for it. Almost free with K8s deployments + HPA.

Limits:

• Database eventually becomes the bottleneck.
• Shared mutable state (cache, queue) still has its own limits.
• Network coordination grows non-linearly past some point.

Y-Axis: Functional Decomposition

Split the monolith into services by domain:

Monolith                    Microservices
┌──────────────┐            ┌─────────┐  ┌─────────┐
│ Orders       │            │ Orders  │  │ Payments│
│ Payments     │     →      └─────────┘  └─────────┘
│ Inventory    │            ┌─────────┐  ┌─────────┐
│ Notifications│            │ Inv.    │  │ Notif.  │
└──────────────┘            └─────────┘  └─────────┘

Benefits:

• Independent deploy cadence.
• Different DBs/runtimes per service.
• Failure isolation (payments outage doesn't kill catalog).

Costs:

• Network latency between services.
• Distributed transactions become Saga + Outbox.
• Operational complexity (deploys, monitoring, tracing).

Rule of thumb: split when teams are larger than ~8 engineers per service, or when domains have different scale/latency profiles.

Z-Axis: Sharding

Same code, different data. Each instance owns a subset (a shard) of the data.

Shard 1: user_id % 4 == 0
Shard 2: user_id % 4 == 1
Shard 3: user_id % 4 == 2
Shard 4: user_id % 4 == 3

Needed when a single DB node can't hold the working set or serve QPS. Common shard keys: user_id, tenant_id, geographic region.

See Sharding & Partitioning for details.

Vertical Scaling

"Bigger box." More CPU, more RAM, faster disk. Limits:

• Cost grows super-linearly past commodity hardware.
• Single failure domain.
• Eventually you hit physical ceiling (largest available SKU).

Right tool for: databases that can't shard, in-memory caches, single-leader systems.

Stateless vs Stateful

Rule: keep the application tier stateless; push state to a dedicated tier that's designed for it.

Read Scaling vs Write Scaling

Different problems:

Read scaling (easier):

• Read replicas.
• Caching (Redis, CDN).
• Materialized views.
• Eventual consistency where acceptable.

Write scaling (harder):

• Sharding.
• Write-behind buffering.
• Batching.
• Split into multiple writable services (e.g., separate write paths for orders vs activity log).

A read-heavy app (10:1, 100:1) is much easier to scale than a write-heavy app.

Amdahl's Law

speedup = 1 / ((1 - P) + P/N)

Where P is the parallelizable fraction. Sobering: even 95% parallel work has a hard ceiling at 20× speedup. The 5% serial bottleneck dominates eventually.

In practice: any single point (a primary DB, a global lock, a coordinator) eventually caps your scale, no matter how many app servers you add.

Universal Scalability Law (Gunther)

Refines Amdahl with coherence cost: as N grows, coordination between nodes pulls throughput down. Plotted, throughput peaks and then declines past an optimum.

Means: more boxes is not always more throughput. Past a point, you're paying more for coordination than you gain in capacity.

Designing for Scale Up-Front

A pragmatic recipe:

1. Make the app stateless. Any future scaling is impossible without this.
2. Externalize sessions. Redis or signed JWTs.
3. Idempotent endpoints. Retry-safe by default (Idempotency-Key).
4. Choose a shardable data model. Even if you start with one DB, design rows around a future shard key.
5. Async where possible. Queue heavy work; don't block request threads.
6. Cache strategically. Identify the 20% hot data; cache that.
7. Measure before scaling. Profile p99, find the bottleneck, scale that layer.

Common Mistakes

[!NOTE] Scale problems are rarely about adding boxes — they're about removing coordination. The fastest single-shard system you can design beats the largest distributed system you can build.

Interview Follow-ups

• "What's the biggest single bottleneck in your design?" — Always have an answer; that's where you'd scale next.
• "Can the DB handle 10× the QPS?" — If no: read replicas + cache for reads; sharding for writes.
• "How would you scale to global users?" — Multi-region deploy, latency-routed DNS / Anycast, regional caches, eventual consistency for non-critical data.

Q: What are SLI, SLO, SLA, and error budgets — and how do they connect?

Answer:

SRE terminology that often confuses engineers. The three are nested:

• SLI: a measurement.
• SLO: an internal target for that measurement.
• SLA: a contract with consequences if you miss the SLO.
• Error budget: how much you can afford to miss the SLO over a window.

SLI — Service Level Indicator

A quantitative measure of one aspect of service quality. Common SLIs:

• Availability: ratio of successful responses to total.

  successful_requests / total_requests
  

• Latency: fraction of requests faster than threshold.

  count(requests where duration < 200ms) / total_requests
  

• Throughput: requests per second.
• Correctness: ratio of responses that match a golden expectation (rare, valuable).
• Durability: probability of data loss over time (mostly for storage).
• Freshness: how stale the data behind a request can be.

SLO — Service Level Objective

A target for an SLI over a window:

99.9% of requests succeed, measured over 30 days
99% of p99 latencies ≤ 200 ms, measured over 7 days

The "9s" budget mapped to downtime:

Anything past 99.99% requires multi-region active-active and disciplined release engineering.

SLA — Service Level Agreement

A contract between you and a customer, often with refund clauses:

"We guarantee 99.9% monthly uptime. Below that, you get 10% credit.
 Below 99% you get 25% credit."

The SLO is internal (your engineering target). The SLA is external (legal). The SLO should be tighter than the SLA — leave headroom for unknowns and customer-friendly fudge.

Error Budget

If SLO = 99.9% availability over 30 days, the error budget is the 0.1% — about 43 minutes of allowed downtime per month.

Used for:

1. Release velocity decisions. Budget remaining → ship aggressively. Budget exhausted → halt risky changes; focus on reliability.

2. Engineering investment. Frequent budget burn → pay down reliability debt. Budget unused → invest in features instead.

                       error budget
   100% ────────────────────────────────  SLO ceiling
        │
        │                ╱╲
        │              ╱    ╲                ← burn
   99.9%├────────────╱────────╲────────────  SLO target
        │          ╱            ╲
        │
        │
   99.5%│
        └────────────────────────────────►  time

Burn Rate Alerts

Better than threshold alerts: alert on how fast you're burning the budget.

Burn rate = (errors_in_window / requests_in_window) / (1 - SLO)

Burn rate = 1   → you'd consume the budget exactly on schedule
Burn rate = 14  → you'd burn 30 days' budget in ~2 days

Common dual-burn-rate alert (Google SRE Workbook):

• Fast: 14× burn rate over 1 hour AND 5 minutes → page.
• Slow: 6× burn rate over 6 hours AND 30 minutes → ticket.

Catches outages quickly without paging on every flap.

Designing SLIs

Good SLI properties:

• User-perceived: measure what users feel, not what your dashboard fancies.
• Computable: from existing telemetry, no manual instrumentation needed.
• Aggregable: by service, by API, by tenant.
• Cheap: doesn't itself cost the budget to compute.

The two go-to SLIs for any HTTP service:

availability = sum(rate(http_requests{status!~"5.."}[5m]))
             / sum(rate(http_requests[5m]))

latency      = sum(rate(http_requests_bucket{le="0.2"}[5m]))
             / sum(rate(http_requests[5m]))

Choosing SLO Targets

Higher targets force expensive architecture (multi-region, sync replication). Don't promise five-nines without buying the engineering.

Decoupling Internal SLO from External SLA

SLA (external):    99.9%
SLO (internal):    99.95%        ← tighter, gives headroom
Budget alert at:   99.97%        ← act before SLO threatened

If you fire alerts at the SLA threshold you've already missed it. Bake in margin.

Where Each Number Lives

SLO ┐
    ├── alert rules (burn-rate based)
    │
    ├── dashboard headline ("we're at 99.91% this week")
    │
    ├── feature freeze trigger (budget < 0)
    │
    └── post-mortem ROI ("how much SLO did this incident cost?")

Common Mistakes

[!NOTE] Error budgets reframe reliability from "always more" to "exactly enough." A team that's never burning its budget should be shipping faster; a team always over budget should slow down and harden.

Interview Follow-ups

• "How do you decide SLO targets for a brand-new service?" — Start with a lenient SLO (99.5%) for a few months, gather data, tighten once stable. Don't promise what you haven't measured.
• "What if a dependency violates its SLO?" — Either renegotiate yours, mitigate via fallback (cache last-known-good), or split usage to a more reliable dependency.
• "What's the difference between MTTR, MTTD, and MTTF?" — Mean Time To Recovery (how long an outage lasts), Detection (how long to notice), Failure (between failures). All three feed into SLO performance.

Q: How do you approach a system design interview — the structured framework.

Answer:

System design interviews are conversational, not technical trivia. The interviewer is testing whether you can drive a 40-minute design conversation, make justified trade-offs, and surface risks. The framework below is what most senior interviewers expect — adapt it, don't recite it.

The 6-Step Framework

1. Clarify requirements      (5 min)
2. Estimate scale            (3 min)
3. API + data model          (5 min)
4. High-level design         (10 min)
5. Deep-dive on 2–3 areas    (15 min)
6. Trade-offs & extensions   (5 min)

Adjust the time per signal from the interviewer — if they push deeper on storage, spend more time on storage.

1. Clarify Requirements

Don't start drawing boxes. Ask:

Functional:

• Core features (pick 2–4; defer the rest).
• Read patterns vs write patterns.
• Multi-tenant? Multi-region?

Non-functional:

• DAU / scale.
• Latency target.
• Consistency requirements (per data class).
• Availability target (SLO).
• Mobile vs web? Geographic distribution?

Verbalize: "I'm assuming we don't need to design the mobile client, just the backend. Sound right?"

This step is graded. Skipping it is a red flag.

2. Estimate Scale

Numbers ground the design. See Back-of-Envelope Estimation.

Compute:

• QPS (read and write separately).
• Storage growth (per day, per year).
• Bandwidth (peak egress).

Match against known limits:

• 50k QPS → cache + replicas.

• 10 TB → sharding.

• 1 Gbps egress → CDN.

3. API + Data Model

API first — what's the contract?

POST   /v1/orders         { items: [...], userId: ... }    → 201 { orderId }
GET    /v1/orders/{id}                                      → 200 { order }
GET    /v1/orders?userId=&cursor=                           → 200 { orders, nextCursor }
PATCH  /v1/orders/{id}    { status: "cancelled" }           → 200

State idempotency: "I'd require Idempotency-Key for POST."

Then the data model — pick a schema and a storage class.

Orders Table (Postgres, sharded by userId)
  id            UUID PK
  user_id       BIGINT  (shard key)
  status        ENUM
  amount        NUMERIC
  created_at    TIMESTAMPTZ
  INDEX (user_id, created_at DESC)

For NoSQL choices: state partition key + sort key + access pattern.

4. High-Level Design

Draw boxes. Standard layers:

Client → CDN → Load Balancer → API Gateway → Service(s) → Cache → DB
                                                       ↓
                                                    Queue → Async Workers

Annotate what each layer does. Common ones:

• CDN: static assets, media.
• LB: L7 (Envoy/HAProxy/Nginx) for HTTP routing.
• API Gateway: AuthN, rate limit, request shaping.
• Stateless services: business logic.
• Cache: Redis for hot reads.
• Primary DB: source of truth.
• Search index: Elastic / OpenSearch.
• Queue/Stream: Kafka / SQS for async.

Tell a request's story end to end. "User clicks 'Order Now' → API gateway → orders service → write to Postgres → publish event to Kafka → notify service → SMS."

5. Deep Dive on 2–3 Areas

Pick the two or three areas where the interview signal is highest. Common deep-dive prompts:

• Sharding: which key, how to rebalance, hot shards.
• Caching: invalidation strategy, eviction, stampede protection.
• Consistency: which operations need linearizability vs eventual.
• Failure handling: what happens if X dies — service, DB primary, region.
• Hot keys: how the system handles a celebrity user / viral post.
• Scaling: where the next bottleneck is and how you'd address it.

Walk through one in detail; let the interviewer steer the others.

6. Trade-offs & Extensions

Self-critique:

• "I'd start without sharding; add it once we cross 5k write QPS."
• "Cache is a write-through pattern; trades latency on writes for stronger reads."
• "We'd lose data in a multi-region disaster; for orders, switch to synchronous cross-region writes once revenue justifies the latency hit."

Mention what you'd add given more time: search, recommendations, monitoring, on-call playbooks.

Anti-Patterns

Signals Interviewers Look For

Senior signals:

• Asks clarifying questions before designing.
• Estimates before optimizing.
• Picks trade-offs deliberately, not by default.
• Names the bottleneck before being asked.
• Talks about operability (deploys, alerts, on-call).
• Pushes back on under-specified asks.

Junior signals:

• Knows specific tech stacks.
• Can draw a typical architecture.

Red flags:

• "We need it consistent, so let's use [strong-consistency DB]" for everything.
• Designing the perfect 100M-user system on minute one.
• Never mentioning failure, observability, or cost.

Useful Phrases

• "For now, let's assume X. We can revisit."
• "I see two options: A trades latency for consistency, B does the opposite. I'd pick A because..."
• "The next bottleneck would be the DB write throughput. To go further: shard by user_id."
• "If a region goes down, here's what happens..."
• "I'd capture this metric and alert on it."

Common Mistakes

[!NOTE] Best preparation: design 10–20 real systems out loud, in 45 minutes each, against a friend or alone. The framework becomes automatic; you spend the interview thinking about trade-offs, not the structure.

Interview Follow-ups

• "What if I gave you another hour?" — Always have a list: observability, capacity planning, deployment story, security threats, cost optimization.
• "What would you NOT do here?" — Show judgment: "I wouldn't use a graph DB for this; the query pattern is key-value."
• "How would you migrate from the existing system?" — Strangler fig pattern; dual writes; feature flag the cutover.

Q: Load balancers — L4 vs L7, algorithms, health checks, gotchas.

Answer:

A load balancer (LB) distributes incoming requests across multiple backend instances. The decision tree starts at OSI layer: L4 (transport) sees IP + port; L7 (application) sees HTTP headers, paths, cookies.

L4 vs L7

L4 for raw throughput, non-HTTP protocols, deep packet workloads. L7 for everything HTTP — that's where the smart routing pays off.

Common Algorithms

For most web services: Power of Two Choices is the modern default. Beats round-robin and least-connections at scale with O(1) memory.

Health Checks

LB must know which backends are alive.

Active: probe each backend on an interval.

• HTTP GET /health every 2–5s.
• Timeout: ≤ 1s typical.
• Mark unhealthy after N consecutive failures (3–5).
• Mark healthy after M consecutive successes (2–3) to avoid flapping.

Passive: observe live traffic.

• Mark unhealthy if outbound error rate spikes.
• Cheaper but slower to react.

Best practice: both — active for fast detection, passive for nuance.

The probe endpoint should reflect what readiness means for traffic. See Health Checks Deep Dive.

Sticky Sessions

Pin one client to one backend across requests. Use when:

• The backend has in-memory session state that isn't replicated.
• Cache warmth matters (consistent hashing).

Mechanisms:

• Source IP affinity (L4): brittle, breaks behind NAT.
• Cookie-based (L7): LB sets/reads a lb-session cookie.
• Consistent hash on key (e.g., user_id in header).

Cost: uneven load when one user spikes. Prefer stateless services + shared cache when possible.

TLS Termination

Two strategies:

A) Terminate at LB (most common)
   client ──TLS──► LB ──HTTP──► backend
   - Cert lives at LB
   - Backend sees plaintext
   - Easy to inspect, route, gzip

B) Passthrough / SNI routing
   client ──TLS──► LB ──TLS──► backend
   - LB only routes by SNI
   - End-to-end encryption
   - No L7 features

Re-encrypt option: terminate, decrypt, then open a new TLS connection to backend. Costs CPU but lets the LB inspect while keeping in-DC traffic encrypted.

Global vs Regional LBs

DNS / Anycast (global)
   │
   ▼
GeoDNS / AWS Global Accelerator
   │
   ▼ routes to nearest region
Regional LB (ALB, NLB, Envoy)
   │
   ▼
Backend instances

Global routing handled by DNS (latency-based, geo-based) or Anycast IP (BGP-based). Regional LB then distributes within the region.

LB as a Failure Domain

The LB is itself a service. Mitigate:

• Multiple LB instances behind a VIP / DNS round-robin.
• Cloud-managed LB: AWS ALB/NLB are themselves multi-AZ by default.
• Self-managed: keepalived + VIP failover, BGP ECMP for active-active.

A single Nginx box is not a load balancer for production.

Service Mesh as Per-Pod LB

In K8s + Istio/Linkerd, each pod has a sidecar that handles client-side load balancing. The "global" LB becomes the cluster ingress; everything inside is mesh-routed.

Benefits:

• Connection pooling per pod.
• Per-call retries, timeouts, mTLS without app code.
• Power-of-two-choices natively.

Cost:

• Sidecar resource overhead.
• More moving parts to debug.

Common Gotchas

Connection Draining

When pulling a backend out (deploy, scale-in), don't kill in-flight connections.

1. Mark backend "draining" — LB stops sending new connections
2. Wait for existing to finish (or hit timeout, e.g. 60s)
3. Kill backend

K8s: combine with preStop hook + terminationGracePeriodSeconds.

LB Architecture Reference

                  Internet
                     │
                     ▼
              ┌──────────────┐
              │ Anycast / DNS │  (global)
              └──────┬───────┘
                     │
        ┌────────────┼────────────┐
        ▼            ▼            ▼
    us-east-1     us-west-2    eu-west-1
        │            │            │
        ▼            ▼            ▼
     ALB/NLB      ALB/NLB      ALB/NLB
        │            │            │
        ▼            ▼            ▼
  K8s ingress  K8s ingress  K8s ingress
        │            │            │
        ▼            ▼            ▼
     pods         pods         pods

Common Mistakes

[!NOTE] A surprisingly large portion of "scaling problems" are LB problems: hot backends, bad sticky sessions, slow health checks. Treat the LB as a first-class component to design, not an afterthought.

Interview Follow-ups

• "How do you load-balance gRPC?" — gRPC uses HTTP/2 long-lived connections; round-robin breaks. Use Envoy with per-request L7 LB, or client-side LB.
• "How do you handle a 'thundering herd' on backend restart?" — Slow-start, jittered staggering, warmup endpoints.
• "How do you do canary deploys via LB?" — Weighted target groups (1% → 10% → 50% → 100%) or header-based routing for a small group of users.

Q: How do CDNs work, and when do you use one?

Answer:

A CDN (Content Delivery Network) is a global network of cache servers (Points of Presence — PoPs) that serve content close to the user, reducing latency and origin load. It's the cheapest single performance optimization a web app can make.

What CDNs Cache

How a Request Flows

1. User in Mumbai requests https://cdn.example.com/img.png
2. DNS → CDN PoP near Mumbai (anycast or geo)
3. PoP looks up img.png in its cache
4. HIT  → return from PoP   (~10 ms latency)
   MISS → fetch from origin (~150 ms first time, then cached for next user)

PoPs sometimes form a tree: edge PoP → regional shield PoP → origin. The shield absorbs origin load by deduplicating misses across many edges.

Cache Key

Defines what counts as "the same object" at the CDN:

default: scheme + host + path + query
   GET /img.png?v=1   ≠   GET /img.png?v=2

Configure:

• Strip query strings if irrelevant.
• Include Accept-Encoding so gzipped vs br vs identity are separate.
• Include Vary: ... headers (CDN respects standards).
• Avoid including auth cookies — turns every user into a unique key.

Cache Headers

Origin tells the CDN how long to cache.

Cache-Control: public, max-age=31536000, immutable
   ↑ public to CDN, browser keeps it 1 year, won't even revalidate

Versioned assets pattern:

/static/app.<hash>.js          # never changes; cache 1 year
/static/app.js                 # mutable; very short cache or no-cache

Revalidation:

Cache-Control: max-age=0, must-revalidate
ETag: "abc123"

On next request, browser/CDN sends If-None-Match: "abc123"
Origin returns 304 Not Modified — saves bandwidth, costs a roundtrip.

Stale-While-Revalidate

Cache-Control: max-age=60, stale-while-revalidate=300

CDN serves cached object for 60s. From 60s–360s, serves stale instantly and asynchronously refetches. Eliminates user-visible refresh latency.

Cache Invalidation (Purge)

Three modes:

• Purge by URL: nuke specific paths.
• Purge by tag (Cloudflare, Fastly): tag assets at insert time, purge by tag — invalidate "all blog posts" with one call.
• Wait for TTL: cheapest; works if TTL is acceptable.

Purge is propagation, not instant — usually < 30s but not zero.

Origin Shield

A "shield" PoP sits between edge PoPs and origin. All misses funnel through the shield.

100 edge PoPs miss "img.png" → 100 requests at origin?
With shield:                  → 1 request at origin (others served from shield)

Critical for protecting origin during traffic spikes.

CDN for Dynamic Content

Two strategies:

1. Cache short.

Cache-Control: public, max-age=10

Even 10 seconds of caching absorbs huge bursts on hot pages (Reddit front page, breaking news). API responses for read-mostly endpoints.

2. Edge compute.

CDN runs your code at PoPs:

• Cloudflare Workers, Vercel Edge Functions, Lambda@Edge, Fastly Compute@Edge.

Use cases: A/B testing, redirects, light personalization, rate limiting, JWT verification. Sub-50ms latency anywhere.

Geographic Routing

DNS-based: CDN's DNS resolver returns IP of nearest PoP based on resolver's location (which is mostly the user's ISP).

Anycast: same IP advertised from many PoPs via BGP; the network routes to the topologically nearest one. Faster failover but harder to debug.

Security Features

Modern CDNs are also security layers:

• DDoS absorption: PoPs absorb L3/L4 floods.
• WAF (web app firewall): rule-based filtering at edge.
• Bot management: CAPTCHA, rate limit by fingerprint.
• TLS termination + cert management: free certs, automatic renewal.
• Origin shield: hide origin IPs.

When NOT to Use a CDN

• WebSockets / SSE (most CDNs are pass-through; some now support).
• Real-time bidirectional protocols.
• Highly personalized HTML that varies per user (cache will miss).
• Internal services not exposed to the internet.

For uncacheable traffic, the CDN still adds value as a TLS terminator + DDoS shield.

Cache Hit Ratio

hit_ratio = hits / (hits + misses)

Targets:

• Static assets: > 95%.
• Dynamic with short cache: 70–90%.
• API caching: varies; measure per-endpoint.

Diagnose low hit ratio:

• Cookies / auth headers in cache key.
• Query string randomization.
• Different Accept headers per client.
• TTL too short.

Costs

Pricing is typically per-GB egress + per-request. Cheaper than your origin's egress. For media-heavy apps, the CDN itself is the dominant cost line — measure and optimize.

Common Mistakes

[!NOTE] The first hour of optimizing a web app's performance is almost always CDN setup: long-lived cached versioned assets, short cached API responses, fingerprinted filenames, edge TLS. Compounding gains for trivial work.

Interview Follow-ups

• "How do you cache user-specific data at the edge?" — Use Vary: Cookie only for the specific cookie that identifies cacheable variants (e.g., locale, not session). Or use Edge Workers to assemble personalized + cached fragments.
• "How do you handle a CDN-cached bad response?" — Tag-based purge or version bump. Treat broken cache as a real outage class.
• "How do you serve video?" — Encode to HLS/DASH segments, signed URLs, long TTL on segments, manifest with shorter TTL. CDN serves the bulk; origin only fingerprints manifests.

Q: REST vs gRPC vs GraphQL — picking an API style.

Answer:

The three dominate modern API design. They optimize for different things: REST for simplicity + caching, gRPC for performance + strict contracts, GraphQL for flexible client queries. Most companies use more than one.

REST

HTTP verbs map to operations on resources, identified by URLs. JSON payload by convention.

POST   /v1/orders                 # create
GET    /v1/orders/123             # read one
GET    /v1/orders?status=open     # list with filters
PATCH  /v1/orders/123             # partial update
DELETE /v1/orders/123             # delete

Strengths:

• Universal tooling (curl, browsers, CDN).
• Cacheable via standard HTTP (Cache-Control, ETag).
• Easy to inspect/debug.
• Versionable by URL or header.

Weaknesses:

• No native streaming (long-polling, chunked, or SSE workarounds).
• Type-loose; relies on OpenAPI/JSON Schema for contracts.
• Multiple round-trips for related data (over-fetching / under-fetching).

gRPC

Defines services in .proto files, generates typed clients/servers in any language. Uses HTTP/2 with binary Protobuf payloads.

service OrderService {
  rpc GetOrder(GetOrderRequest) returns (Order);
  rpc ListOrders(ListOrdersRequest) returns (ListOrdersResponse);
  rpc WatchOrders(WatchRequest) returns (stream OrderEvent);   // server stream
  rpc UploadOrders(stream Order) returns (Summary);            // client stream
  rpc Chat(stream Msg) returns (stream Msg);                   // bidi
}

Strengths:

• Strong typing, generated code.
• Binary format → smaller, faster than JSON.
• Native streaming (server, client, bidirectional).
• HTTP/2 multiplexing — many calls one connection.
• First-class deadlines, cancellation, metadata.

Weaknesses:

• Browser support requires gRPC-Web proxy (Envoy or in-process).
• Less universal tooling (specialized debuggers needed).
• Schema changes require .proto distribution to consumers.

Best for: service-to-service internal APIs, mobile apps with native clients, high-throughput RPC.

GraphQL

A query language. Clients send a single query asking for exactly the fields they need across multiple resources.

query {
  order(id: 123) {
    id
    total
    customer {
      name
      email
    }
    items {
      product { name price }
      quantity
    }
  }
}

Strengths:

• Single request fetches a deep object graph.
• Client picks fields — no over-fetching.
• Self-documenting via introspection.
• Strong typing via SDL.

Weaknesses:

• Caching is harder (no URL-keyed cache; needs persisted queries + custom cache layer).
• N+1 query risk on the server (mitigated by DataLoader-pattern batching).
• Authorization is per-field (more surface area).
• "Resolver explosion" — many code paths to maintain.

Best for: aggregating data from many services for a single UI; mobile apps wanting minimal payloads; partner APIs where flexibility matters.

Comparison Table

Versioning

REST: /v1/orders, /v2/orders — old and new run side by side.

gRPC: Protobuf evolution rules. Add fields with new tag numbers; never reuse or change types. Old clients ignore unknown fields.

GraphQL: never version; deprecate fields with @deprecated, add new ones. The schema evolves like an org chart.

Error Models

REST: HTTP status codes + JSON body.

• 4xx client error, 5xx server error.
• RFC 7807 "Problem Details" for structured errors.

gRPC: status codes (OK, INVALID_ARGUMENT, NOT_FOUND, ABORTED, ...).

• Rich error model via google.rpc.Status with details.

GraphQL: always 200 OK with errors array in body. Has to define partial-success.

Streaming Comparison

When to Pick Each

Internal service-to-service          → gRPC
Public REST API for many consumers   → REST + OpenAPI
Mobile app with constrained network  → GraphQL or gRPC (compact payloads)
Browser fetch, hit-and-run           → REST
Real-time bidirectional              → gRPC or WebSocket
Aggregation across many backends     → GraphQL (BFF pattern)

Hybrid Architectures Are Normal

Browser ──REST/GraphQL──► Edge / BFF ──gRPC──► Services ──gRPC──► Services
                                                │
                                                ▼
                                              DB / cache

• REST or GraphQL on the edge for browser ergonomics.
• gRPC internally for performance and contracts.

Pagination

REST: ?cursor=...&limit=... (cursor-based) or ?page=...&size=... (offset-based).

gRPC: PageToken pattern.

GraphQL: Relay Cursor Connection spec — connections, edges, page info.

Cursor-based scales; offset-based breaks past a few thousand rows.

Idempotency

POST is non-idempotent by default. All styles handle this the same way:

• Client sends Idempotency-Key header / metadata.
• Server stores result keyed by it; replays on retry.

See Idempotent REST APIs.

Common Mistakes

[!NOTE] Pick by who consumes your API and what they need. Internal microservices = gRPC. Diverse external consumers = REST. UI aggregating lots of data = GraphQL. There is no general winner.

Interview Follow-ups

• "How do you do auth in gRPC?" — Metadata headers (often a JWT). Interceptors verify per-call. mTLS for service-to-service.
• "What's HATEOAS and do you use it?" — Hypermedia in REST responses (links to next actions). Theoretically elegant; rarely worth the complexity in practice.
• "How does GraphQL handle file uploads?" — Multipart spec; usually offload to a separate REST endpoint that returns a URL.

Q: WebSockets vs Server-Sent Events vs Long Polling — picking real-time transport.

Answer:

Three ways to push data from server to client. Each picks a point on the bidirectionality × overhead × infrastructure-cost triangle.

Long Polling

Client makes HTTP request; server holds it open until data arrives (or timeout), then responds. Client immediately reopens.

client ──GET /events──► server (holds for 30s)
client ◄──response─────  server (data arrives or timeout)
client ──GET /events──► server (next poll)

Pros:

• Plain HTTP. Works through proxies, firewalls, CDN.
• No new protocol.

Cons:

• Connection setup overhead per cycle.
• Server holds many idle connections.
• Latency = connection-reopen time.

Use for: legacy environments where WS / SSE aren't supported.

Server-Sent Events (SSE)

Single long-lived HTTP/1.1 or HTTP/2 connection. Server sends a stream of text events.

GET /stream HTTP/1.1
Accept: text/event-stream

Response:
HTTP/1.1 200 OK
Content-Type: text/event-stream

data: {"type": "msg", "body": "hi"}\n\n
data: {"type": "msg", "body": "hello"}\n\n
...

Pros:

• One direction (server → client) — perfect for feeds, notifications.
• Auto-reconnect built into browser API.
• Plain HTTP — proxies, CDN edge support.
• Built-in id for resume.

Cons:

• One-way.
• Text only (binary needs base64).
• Some legacy proxies buffer SSE responses.

Use for: live notifications, server logs, dashboard updates, AI chat streaming.

WebSockets

Persistent bidirectional connection over a single TCP socket. Initiated via HTTP Upgrade.

GET / HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: ...

← HTTP/1.1 101 Switching Protocols

After upgrade, the socket carries WS frames in either direction.

Pros:

• True bidirectional.
• Low overhead per message (few bytes).
• Binary or text.

Cons:

• Stateful — server must track each connection.
• LB / CDN support varies (sticky required).
• Reconnect logic non-trivial.

Use for: chat, collaborative editing, gaming, trading platforms.

Comparison

Connection Cost Per Server

Each connection consumes:

• OS file descriptor (1).
• TCP socket buffers (~64 KB default).
• Application memory (KB to MB depending on state).

Typical server: 100k concurrent connections (8 GB RAM allocated)
Tuned (epoll/io_uring, lean app): 1M+ connections

Scale-out: shard connections across N gateways; LB by client ID or sticky hashing.

Choosing in 30 Seconds

Need bidirectional? ─── yes ──► WebSocket
              │
              no
              │
Server pushes events only? ─── yes ──► SSE
              │
              no
              │
Don't care about latency, just need updates? ─── Long Poll

Real-Time at Scale Patterns

Whichever transport, the backend looks similar:

event source ──► message bus (Kafka/Redis pub-sub) ──► connection gateway ──► clients

gateway:
  - holds N connections
  - subscribes to topics relevant to its connected users
  - filters and pushes events to each client

Sticky LB → user always lands on the same gateway → that gateway holds the subscription.

Heartbeats / Keepalives

All three need application-level keepalives:

• WS: send ping/pong every 30s.
• SSE: server sends comment :keepalive\n every 15s.
• LP: timeout the long-poll at 30–60s; client reconnects.

Without heartbeats, idle connections die silently behind NATs / proxies.

Auth

• WS: usually JWT in URL query param or sub-protocol header (cookies work too).
• SSE: same-origin cookies or Authorization header (no headers in browser EventSource; use cookies or query token).
• LP: standard cookie / header auth.

Token rotation: send signed short-lived token via WS / SSE message; refresh before expiry.

Backpressure

If client can't keep up:

• WS: app reads from socket; OS buffers fill; producer blocks or drops.
• SSE: similar; some servers drop oldest.
• LP: client controls pace (next poll happens after processing).

Server policy: queue per-connection (bounded), drop oldest on overflow, or disconnect slow consumers.

CDN / Edge

• WS: limited; some edges (Cloudflare, Fly.io) support; CloudFront does not natively.
• SSE: HTTP-native; most CDNs work if buffering disabled.
• LP: native HTTP; full CDN support.

Common Mistakes

[!NOTE] Default to SSE for "server → client updates" use cases — it's HTTP, simple, infra-friendly. Reach for WS only when you need bidirectional. Long-poll is legacy.

Interview Follow-ups

• "How would you scale WebSockets to 10M concurrent?" — Shard gateways; per-user sticky LB; pub/sub bus (Redis or Kafka) for per-user routing.
• "How is WebRTC different?" — P2P data/media; needs signaling (often WS); used for low-latency media (voice/video).
• "WebTransport?" — Modern alternative to WS over HTTP/3 (QUIC); multiplexed streams, unreliable datagrams. Not widely adopted yet.

Q: API Gateway and BFF (Backend for Frontend) — what role do they play?

Answer:

In a microservices world, clients can't talk to 100 backend services directly. The API gateway is the front door: one stable endpoint that routes, authenticates, rate-limits, and aggregates. The BFF is a specialization: one gateway per client type (web, iOS, Android).

API Gateway

                       ┌──────────────┐
client ───────────────►│ API Gateway  │
                       └──────┬───────┘
                              │
        ┌─────────────┬───────┼─────────┬───────────┐
        ▼             ▼       ▼         ▼           ▼
     Orders      Inventory  Search    Payments   Users
     Service     Service    Service   Service    Service

Responsibilities:

• Routing: URL/path/host → service.
• AuthN / AuthZ: verify JWT; map to user context.
• Rate limiting: per-IP, per-API-key, per-tenant.
• Request shaping: header injection, request transformation.
• Response shaping: aggregate, filter.
• Caching: HTTP-cache headers for read-mostly endpoints.
• Observability: per-route metrics, traces.
• Circuit breaking / retries (optional).

Popular implementations: AWS API Gateway, Kong, Envoy, Tyk, Apigee, custom-built.

BFF Pattern

Different clients need different shapes of data. Rather than one gateway serving all, one BFF per client:

                    ┌──── Web BFF      ─► Services
                    │
client (web) ───────┘

                    ┌──── Mobile BFF   ─► Services
                    │
client (iOS) ───────┘

                    ┌──── Partner BFF  ─► Services
                    │
client (partner)────┘

Each BFF:

• Owned by the client team (closer to UI requirements).
• Optimized for that client (smaller payloads on mobile; richer on web; restricted on partner).
• Aggregates backend services into one client-friendly response.

GraphQL is often the right tool for the BFF layer — clients pick what they need.

Why Not "Just Have Services Call Each Other"

Pre-gateway client-direct era problems:

• Clients hardcoded N service URLs.
• Each service handled auth differently.
• Cross-cutting concerns (rate limit, retries) duplicated.
• Browser CORS hell.

Gateway centralizes that. Worth the extra hop.

Anti-Patterns

Aggregation

GET /v1/pages/order/123
   ├── parallel call: orders.get(123)
   ├── parallel call: payments.byOrder(123)
   ├── parallel call: shipping.byOrder(123)
   ├── parallel call: users.get(order.user_id)
   └── compose response

Gateway/BFF orchestrates; one client request → N parallel service calls → one merged response.

Auth at the Gateway

Standard pattern:

client → gateway
   gateway: verify JWT signature + expiry
   gateway: extract claims → set X-User-Id header
   gateway: forward to service

service: trust X-User-Id from gateway (internal network)

Service shouldn't re-verify (slow); but gateway must be authoritative.

mTLS between gateway and services prevents spoofing internal headers.

Rate Limiting

Multi-level:

• Per-IP at edge.
• Per-API-key / per-token at gateway.
• Per-user / per-tenant at gateway or BFF.

See Rate Limiting Algorithms.

Versioning

Gateway-level versioning lets you migrate services behind it:

/v1/orders → orders-service-v1
/v2/orders → orders-service-v2

Or header-based:

Accept-Version: 2

Gateway as Performance Layer

• Compression (gzip/br).
• HTTP/2 multiplexing to upstream.
• TLS termination centralized.
• Edge caching for read-mostly responses.
• Connection pooling to backends.

Failure Modes

Common Mistakes

[!NOTE] The gateway is your platform's contract with the outside world. Keep its responsibilities narrow (routing, auth, rate-limit, basic shape); push everything domain-specific into services or BFFs.

Interview Follow-ups

• "Where do you put logging / tracing?" — Gateway emits per-request entries; trace context propagated via headers (W3C traceparent) to all downstream calls.
• "How is API Gateway different from a service mesh?" — Gateway = north-south traffic (client → cluster). Mesh = east-west (service ↔ service). Different layers; often coexist.
• "How do you avoid the gateway becoming a SPOF?" — Multi-region, multi-AZ; managed (cloud) when possible; client retries with backoff.

Q: SQL vs NoSQL — when to pick which?

Answer:

"SQL vs NoSQL" is a misleading split — there are at least five NoSQL families with very different properties. The right question is "given my access patterns and constraints, which data model and consistency story fits?"

The Families

The Right Mental Model

Don't pick one for the whole system. Polyglot is the norm:

Source of truth:    Postgres        (orders, users — ACID)
Hot reads:          Redis            (sessions, leaderboards)
Search:             Elasticsearch    (full-text)
Analytics:          ClickHouse       (rollups)
Object/blob:        S3                (uploads)
Event bus:          Kafka             (audit, fanout)
Time-series:        Timescale         (metrics)

Pick the source of truth carefully; derived stores can be rebuilt.

When SQL Is Right

• Strong consistency required (financial, regulated, identity).
• Complex queries (joins, aggregates) you can't predict in advance.
• Mature tooling (migrations, backups, monitoring) matters.
• Data fits in one logical DB (under a few TB hot working set).
• You want referential integrity enforced.

Postgres is the boring, correct answer for ~80% of new services.

When NoSQL Is Right

Each family for different reasons:

Document store (Mongo, Dynamo):

• Schema evolves rapidly.
• Object naturally hierarchical (a "product" with nested specs/reviews).
• Single-record reads dominate.
• Multi-tenancy with isolated schemas per tenant.

Wide-column (Cassandra):

• Write throughput >100k QPS that one RDBMS can't deliver.
• Time-series or log-like data partitioned by entity + time.
• Multi-region active-active with tunable consistency.

Key-value (Redis):

• Lookups by exact key.
• TTL-based eviction (sessions, rate-limits).
• Low-latency requirements (sub-millisecond).

Graph DB:

• Traversals more than 2–3 hops are common.
• "Friends of friends," fraud rings, recommendation graphs.
• (Often: Postgres + recursive CTE handles 2-hop just fine.)

Search:

• Full-text, faceted search.
• Ranking by relevance, not just filtering.

Time-series:

• Append-only metrics.
• Range queries on timestamp.
• Compression of similar adjacent values.

Schema Flexibility

A common reason teams reach for NoSQL: "we don't know the schema yet."

Reality:

• Postgres supports JSONB columns with indexes — gives you schemaless with optional structure.
• "Schemaless" doesn't mean "no schema" — it means the schema is in app code, harder to audit.
• Real schema evolution still needs care.

Pattern: Postgres with JSONB for the flexible bits, typed columns for the well-defined bits. Avoid full document-DB unless you really need it.

Access Pattern Matters More Than Tech

NoSQL forces you to design for access patterns up front (DynamoDB famously). SQL lets you write any query and add an index later.

Example: a "user's recent orders" page.

SQL (Postgres):
  SELECT * FROM orders WHERE user_id = ? ORDER BY created_at DESC LIMIT 20;
  index: (user_id, created_at DESC)

DynamoDB:
  PK = user_id, SK = created_at
  Query(PK = u123, SK desc, limit 20)

Same access pattern, both work. SQL gives you GROUP BY status for free; DynamoDB needs a second index.

Joins

A common claim: "NoSQL doesn't do joins." Mostly true. The reply is denormalize:

• Embed related data in the document.
• Maintain a materialized view.
• Pre-compute the join offline.

If your model needs joins on every read, SQL is the lower-friction choice.

Transactions

If your business logic crosses many entities atomically, prefer SQL.

Scaling Story

"Scales horizontally out of the box" was the original NoSQL appeal. Modern SQL has caught up — distributed SQL is no longer rare.

Cost & Operability

• Managed SQL (RDS, Aurora, Cloud SQL): expensive at scale, very mature ops.
• Managed NoSQL (DynamoDB): pay per request + storage; cheap at low scale, expensive at extreme scale.
• Self-hosted: cheaper compute but you own the on-call.

DynamoDB's "auto-scaling" is real but bills can surprise. Cassandra is brutal to run yourself.

Common Mistakes

Decision Heuristics

• Default: Postgres for source of truth. Add specialized stores as patterns emerge.
• Write QPS > 50k: think Cassandra/DynamoDB or Postgres with Citus.
• Search: don't do LIKE '%x%' in Postgres at scale — use Elasticsearch.
• Sub-millisecond reads: Redis in front of the DB.
• Multi-region active-active: Cassandra, Cosmos, or Spanner/Cockroach.
• Data > 100 TB hot: distributed SQL or columnar warehouse.

[!NOTE] The most common production stack is Postgres + Redis + S3 + a queue. That's enough for most consumer-scale apps. Reach for more exotic stores only when access patterns demand it.

Interview Follow-ups

• "You said Postgres. How would you scale it past one machine?" — Read replicas; partition table by time/customer; eventually shard with Citus or Vitess; or migrate to distributed SQL.
• "Why is DynamoDB schemaless?" — Schema is enforced at app level; tables only require PK (and optional SK). Trades validation for flexibility.
• "What's HTAP?" — Hybrid Transactional/Analytical Processing — single DB serving OLTP + analytics. SingleStore, TiDB, Spanner. Niche; usually splits via CDC are simpler.

Q: Database indexing — B-tree, hash, GIN, covering, composite. Trade-offs.

Answer:

Indexes turn O(N) table scans into O(log N) lookups. Each one costs disk space + write amplification. Understanding which index helps which query is the difference between a 5 ms response and a 5 second one.

B-Tree (default)

                [50, 80]
               /    |   \
        [20,30,40] [60,70] [85,90,99]

Sorted, balanced tree. Logarithmic depth. Used by:

• Postgres (default)
• MySQL InnoDB primary + secondary
• Most RDBMS

Supports:

• Equality: =
• Range: <, >, BETWEEN
• Prefix wildcard: LIKE 'abc%' (not '%abc')
• ORDER BY matching index direction

Doesn't help:

• Functions: WHERE lower(email) = ? (use functional index)
• Suffix wildcards: LIKE '%abc' (use trigram or reverse index)
• <> (not equal) — index scanned but rarely useful

Hash Index

hash(key) → bucket → entry

Pros: O(1) lookup. Cons:

• No range queries.
• No ordering.

Postgres has hash indexes but rarely worth it over B-tree. Used mostly internally (e.g., in-memory hash joins).

Composite Index

Index on multiple columns, ordered:

CREATE INDEX ON orders (user_id, created_at DESC);

Helps queries that filter by the leftmost prefix:

WHERE user_id = ?                              -- ✅ uses index
WHERE user_id = ? AND created_at > ?           -- ✅
WHERE user_id = ? ORDER BY created_at DESC     -- ✅ no sort needed
WHERE created_at > ?                           -- ❌ doesn't help

Order matters. Lead with the most selective column you filter on.

Covering Index (Index-Only Scan)

If the index includes all columns a query needs, the DB never reads the table:

CREATE INDEX ON orders (user_id, created_at) INCLUDE (status, total);

SELECT user_id, created_at, status, total
FROM orders
WHERE user_id = ?;
-- index-only scan; skip table

Postgres INCLUDE and MySQL InnoDB cluster keys give this.

Partial Index

Index a subset of rows:

CREATE INDEX ON orders (created_at) WHERE status = 'pending';

Smaller, faster for queries that match the WHERE. Useful for hot subsets (pending jobs, active users).

Functional / Expression Index

CREATE INDEX ON users (lower(email));

SELECT * FROM users WHERE lower(email) = ?;     -- uses index

Needed when the query applies a function to a column. Without it, the function runs per row.

GIN (Generalized Inverted Index)

For multi-valued columns: arrays, JSONB, text-search.

CREATE INDEX ON orders USING gin (tags);
SELECT * FROM orders WHERE tags @> ARRAY['urgent'];

CREATE INDEX ON docs USING gin (to_tsvector('english', body));
SELECT * FROM docs WHERE to_tsvector('english', body) @@ to_tsquery('postgres');

Each tag/word becomes an inverted-index entry pointing to rows containing it.

GIST (Generalized Search Tree)

Tree-based, supports complex types:

• Geometry (PostGIS).
• Range types.
• Full-text search (older).

CREATE INDEX ON places USING gist (location);
SELECT * FROM places WHERE ST_DWithin(location, ?, 1000);

BRIN (Block Range INdex)

Tiny index storing per-block summary stats (min/max). Useful for big tables where rows are naturally ordered (timestamps):

CREATE INDEX ON events USING brin (created_at);

Small index, helps range scans on time-ordered tables. Bad for random-order data.

Cost of an Index

Every index adds:

• Disk space (~10–30% of table).
• Write amplification: inserts and updates must update all relevant indexes.
• Memory: cached pages in the buffer pool.

Indexes are not free. Don't index columns that aren't filtered/sorted.

When NOT to Index

• Low-cardinality columns (gender, status with 3 values). Postgres planner often prefers seq scan.
• Frequently updated columns (especially in hot tables).
• Tables < ~10k rows (seq scan is faster anyway).

Reading Query Plans

Postgres:

EXPLAIN ANALYZE SELECT * FROM orders WHERE user_id = 42;

Look for:

• Index Scan (good).
• Index Only Scan (best).
• Seq Scan on a big table (smell).
• Bitmap Heap Scan (often fine for multi-row results).
• Sort step (consider an index that returns pre-sorted).

Index Maintenance

• REINDEX: rebuild bloated indexes (Postgres has REINDEX CONCURRENTLY since 12).
• Statistics: ANALYZE updates table statistics; planner uses them.
• Auto-vacuum: tune for write-heavy tables.

Sharded Indexes

In sharded systems:

• Local index: each shard's own index. Fast for queries that include shard key.
• Global secondary index: separate store keyed by non-shard column; cross-shard queries → fan out vs centralized index.

DynamoDB GSI is the latter; Cassandra secondary index is the former.

Common Mistakes

[!NOTE] The most useful skill is reading EXPLAIN ANALYZE. Indexes are not magic; the planner picks one only if it thinks it'll help. Verify, don't assume.

Interview Follow-ups

• "How would you index WHERE status = 'active' AND created_at > ??" — Partial index on (created_at) WHERE status = 'active'; or composite (status, created_at).
• "How does MySQL InnoDB primary key clustering differ from Postgres?" — InnoDB stores rows physically in PK order — primary key lookup is the table. Postgres stores rows in heap; index is separate.
• "When would you choose hash over B-tree?" — Equality-only queries with very high cardinality and no range needs; in practice, rarely.

Q: Sharding & partitioning — strategies, key choice, rebalancing, gotchas.

Answer:

Partitioning is splitting a dataset into subsets. Sharding is partitioning across separate machines. You shard when one machine can't hold the working set, serve the QPS, or survive the disk-write load.

The Three Strategies

Range Partitioning

shard A: keys [aaa-bbb)
shard B: keys [bbb-ccc)
shard C: keys [ccc-zzz)

Used by HBase, BigTable, MongoDB (range mode), CockroachDB. Splits/merges shards automatically as data grows.

Strength: scan WHERE key BETWEEN x AND y reads one (or few) shards. Weakness: monotonically increasing key (e.g., timestamp) → all writes to last shard.

Hash Partitioning

shard = hash(user_id) % N

Spreads writes uniformly. Used by Cassandra, DynamoDB, MySQL (by app), Redis Cluster.

Strength: even load (if keys are well-distributed). Weakness:

• No WHERE key BETWEEN ... range queries — scatter-gather across all shards.
• Adding shards changes % N → almost all keys move. (See: consistent hashing.)

Consistent Hashing

Place shards and keys on a ring 0..2^64. Each key maps to the next shard clockwise. Adding/removing one shard moves only 1/N of the keys.

       0
        ╲
         ●─── shard A (token 100)
       ╱
    keys 50..100  → A

    keys 100..220 → B
         ●─── shard B (token 220)
       ╱
   ...

Refinement: each shard owns multiple virtual nodes (tokens) on the ring → smoother rebalance. Cassandra uses 256 vnodes/server by default.

See Consistent Hashing for the full mechanics.

Choosing a Shard Key

The most consequential decision. Bad shard keys cause:

• Hot shards (one shard takes 90% of load).
• Scatter-gather queries (every read hits every shard).
• Cross-shard transactions (slow, complicated).

Good shard key properties:

• High cardinality (millions+ of distinct values).
• Uniform access pattern (no celebrity hot keys).
• Co-locality: rows accessed together share a shard.
• Stable over time (not monotonically increasing).

Common keys:

Bad keys:

• created_at (monotonic → hot shard).
• country (skewed; US dominates).
• status (low cardinality).
• Mutable fields (key change = data movement).

Composite Shard Keys

Combine fields for cardinality + locality:

shard_key = hash(tenant_id) + range(created_at)

Hash spreads load across shards; range within a shard keeps a tenant's recent data together. Used by MongoDB's compound shard keys and DynamoDB's PK + SK design.

Hot Shard Problems

Symptoms: one shard at CPU/disk limit while others idle.

Causes:

1. Skewed key distribution (one tenant huge).
2. Sequential key (timestamp, auto-increment ID).
3. Hot row within a shard (celebrity user).

Mitigations:

• Salt the key: prepend a random small bucket → bucket_n_userid. Decompose on read with scatter-gather.
• Re-key by composite: hash a more uniform field.
• Split the hot shard (range partitioning supports this automatically).
• Tiered access: separate read replicas just for the hot tenant.

Rebalancing

Adding capacity needs to move some data without taking the system offline.

Production systems: vnode + background streaming. Cassandra, ScyllaDB, DynamoDB do this transparently.

Cross-Shard Queries

Queries that span multiple shards are slow:

SELECT * FROM orders WHERE customer_email = ? ; -- shard key is user_id

Options:

1. Secondary index on email → maintained per shard, queried via scatter-gather.
2. Global secondary index → separate index store keyed by email (writes are now cross-shard).
3. Application-level fanout → query all shards in parallel, aggregate.

For aggregate queries (counts, sums), use a separate analytics store (ClickHouse, Druid).

Cross-Shard Transactions

Hard. Options ordered by complexity:

1. Avoid them. Co-locate via shard key design.
2. Saga — distributed multi-step workflow with compensation. See Saga Pattern.
3. 2PC — synchronous, blocking, complex. Last resort.
4. Use a distributed SQL DB (Spanner, CockroachDB, YugabyteDB) that does it for you.

Schema Per Shard

Each shard typically has the same schema. Migrations must apply to all shards — usually orchestrated by a tool (Vitess, Citus) or careful application code.

Multi-tenant per-customer-DB pattern (one DB per tenant) trades shard-management complexity for migration complexity — each new schema change is N migrations.

Examples by System

Common Mistakes

[!NOTE] Sharding is a one-way door. Pick the key with care; getting it wrong costs months of migration. Better to design for shardability and not shard than to shard with the wrong key.

Interview Follow-ups

• "How would you migrate from one shard key to another?" — Dual write to old + new (mapped by both keys); backfill old data; switch reads; remove old after burn-in.
• "How do range queries work on hash-partitioned data?" — They don't, efficiently. Use a separate index (search engine, materialized view) keyed appropriately.
• "What's the failure mode of consistent hashing under node loss?" — Node's range is taken over by its neighbor → load doubles. With vnodes, load spreads across many neighbors.

Q: Replication topologies — single-leader, multi-leader, leaderless. Trade-offs.

Answer:

Replication maintains copies of data on multiple machines for availability, durability, and read scaling. The three topologies — single-leader, multi-leader, leaderless — each pick a different point in the consistency / availability / latency space.

Single-Leader (Primary-Replica)

        writes
client ───────► leader ───────► follower
                  │  ───────► follower
                  │
                async or sync replication

• All writes go to one leader.
• Followers receive a stream of changes (binlog, WAL).
• Reads can go to followers (eventual) or leader (strong).

Used by: Postgres, MySQL, MongoDB (replica set), most managed RDBMS.

Sync vs async replication:

• Async: leader acks write before followers apply. Fast. Risks data loss if leader dies before replication.
• Sync to one: at least one follower must ack. Safer, slower.
• Sync to majority (Raft-style): durable but cross-AZ latency tax.

Failover:

• Detect leader failure.
• Promote most up-to-date follower.
• Reroute clients.

Window where writes are lost = unreplicated lag at moment of crash.

Multi-Leader (Active-Active)

       writes
client ───► leader A  ◄───► leader B  ◄─── client (other region)
                ▲                ▲
                └───── replicate ─┘

Multiple leaders, often one per geographic region. Each accepts writes locally; replicates async to others.

Used by: BDR (Postgres), CouchDB, multi-region MySQL with custom plumbing, Cassandra (per-DC), Cosmos DB multi-write.

Trade: local-region latency, but write conflicts between leaders. Same key updated in A and B concurrently → which wins?

Conflict resolution:

• Last-write-wins (LWW): timestamp-based; can drop data.
• App-defined merge: e.g., union for sets, max for counters.
• CRDTs (G-Counter, OR-Set): merge correctly by construction.
• Manual: store both, raise to user (rare).

Leaderless

                  ┌─► node A
client ──────────►├─► node B
                  └─► node C

Writes go to multiple nodes in parallel. Reads also go to multiple nodes and the client (or coordinator) picks the latest version (often via timestamps).

Used by: Cassandra (default), DynamoDB, Riak, ScyllaDB.

Quorum math (N replicas, W write acks needed, R read acks needed):

W + R > N    → strong consistency for that op
W + R ≤ N    → eventual

Common: N=3, W=2, R=2 (read/write quorum).

Read repair / anti-entropy:

• Read-time repair: when read returns mismatched versions, write the latest back to lagging replicas.
• Merkle-tree-based background sync.
• Hinted handoff: temporarily store writes for an unreachable replica.

Sync vs Async Tradeoffs

Replication Lag

For async or leaderless reads, the follower can be behind. Implications:

You write to leader: balance = 100
Immediately read from follower: balance = 90 (lag of 1 second)

Patterns to handle:

• Read-your-writes: route reads to the leader for the user's recent writes (sticky for a window).
• Monotonic reads: pin the user to one follower so reads don't bounce between replicas at different lags.
• Bounded staleness: SLA on max lag; alert if exceeded.

Failover and Split Brain

When a leader is unreachable, who promotes?

• Manual: ops team flips the switch. Slow but safe.
• Automated with consensus: Raft/Paxos for promotion. Used by etcd, Consul, modern Postgres (Patroni).
• STONITH (Shoot The Other Node In The Head): kill the old leader to prevent two-leader scenarios.

Split brain: network partition makes each side think it's the leader. Both accept writes → divergence. Mitigations:

• Require quorum to be leader.
• Fence the minority side via STONITH.
• Use a coordination service (etcd) for leases.

Geographic Replication

For multi-region, choose:

Replicated Logs

The underlying mechanism for most replication is a replicated log (WAL, binlog, Raft log, Kafka). Followers consume the log and apply changes in order.

Why logs:

• Order is unambiguous.
• Replay is idempotent (LSN-based).
• Lag is observable (offset distance).
• Recovery = "catch up the log."

Backups vs Replicas

Replicas are not backups. Both protect against hardware failure. Only backups protect against logical errors (bad query deleted everything; replica deleted it too).

Standard: replicas for availability, snapshots + point-in-time recovery for logical disasters.

Common Mistakes

[!NOTE] Pick a replication topology by the question "what should happen during a partition?" Single-leader CP loses availability. Leaderless AP loses recency. Multi-leader trades both for latency. There's no free combination.

Interview Follow-ups

• "How would you do zero-downtime failover?" — Streaming replication + auto-promotion + connection draining + retry on the client side.
• "What is semi-sync replication?" — Leader waits for at least one follower's ACK before responding to client. Bounded data-loss window.
• "When is multi-leader the right call?" — Multi-region active-active where local-write latency matters AND your data tolerates merge / LWW (calendars, document collaboration, social).

Q: Object storage (S3) — when, what guarantees, gotchas.

Answer:

S3-like object storage is the cheapest, most durable way to keep blobs at scale. It's not a filesystem — it's an immutable key-value store with HTTP API.

What It Is

Bucket: my-bucket
  Key: photos/2025/04/img1.jpg → bytes
  Key: photos/2025/04/img2.jpg → bytes
  Key: backups/db_20250401.sql.gz → bytes

Flat namespace. Slashes in keys are just characters; folder prefixes are a convention.

Properties

• Durability: 11 9s (S3 standard). Multiple copies across AZs.
• Availability: 99.99% (S3 standard).
• Strong read-after-write consistency (since Dec 2020 for S3).
• Cost: cheap storage (~$23/TB/month standard), per-request fees, egress is expensive.
• Throughput: 3,500 PUT/copy/delete and 5,500 GET per second per partition prefix.

Storage Classes

Move data via lifecycle policies (e.g., "after 30 days → IA, after 1 year → Glacier").

What Object Storage Is GOOD For

• Large files (photos, videos, backups).
• Static website assets.
• Data lake (Parquet, ORC) for analytics.
• Logs / archives.
• ML training data sets.
• Software releases / installer files.

What It's BAD For

• Frequent updates to the same object (no in-place modify; full rewrite).
• Tiny files (per-request overhead dwarfs payload).
• Low-latency reads (~10–100 ms per GET vs microseconds for Redis).
• Filesystem semantics (no rename — just copy + delete).

Multipart Upload

Large files split into parts (5 MB – 5 GB each):

1. Initiate multipart upload → get UploadId.
2. Upload parts in parallel.
3. Complete (with list of parts + ETags).

Resume: re-upload only failed parts. Cap: 10,000 parts → max object 5 TB.

Presigned URLs

Time-limited URL granting access without sharing credentials:

url = s3.generate_presigned_url('put_object',
    Params={'Bucket': 'b', 'Key': 'k'}, ExpiresIn=300)
# client uploads directly to S3 with this URL

Use for direct browser uploads (skip your backend); secure media delivery.

Versioning

Enable on bucket → every PUT creates a version. Delete = adds a "delete marker"; old versions remain until explicitly removed.

Protects against accidental overwrites/deletes. Cost: storage of all versions.

Object Lock / Compliance

Write-Once-Read-Many. Object can't be deleted for a retention period (governance or compliance mode).

Used for audit logs, regulatory archives, ransomware protection.

Cross-Region Replication

Bucket-level: every object replicated to another region (sync or async).

Used for DR or for serving traffic close to users.

Access Patterns

Read patterns:

• Direct from client via presigned URL (most common).
• Through CDN (CloudFront) for caching + edge.
• From backend (analytics jobs).

Write patterns:

• Direct from client via presigned URL.
• From backend (uploads, ETL).

For browser apps: presigned URL + CDN. Backend rarely intermediates the bytes.

Cost Optimization

• Lifecycle policies → cold tier old data.
• Intelligent-Tiering for unpredictable access.
• Egress is the most expensive line — use CloudFront / Cloudflare R2 for free egress.
• Avoid LIST over millions of keys; use S3 Inventory reports.

Eventual vs Strong Consistency

Pre-Dec-2020 S3 was eventually consistent for reads after writes. As of now: strong read-after-write consistency for all operations.

You can read your write immediately. Same for delete.

Common Pitfalls

S3 Event Notifications

PUT object in bucket → SNS / SQS / Lambda trigger

Common patterns:

• Trigger image-resize on upload.
• Index new files into Elasticsearch.
• Notify users of new content.

Multi-Region Architecture

client uploads → nearest region's bucket
            │
            ▼ async replicate
       other region buckets

Read from nearest region; cross-region reads possible but slower.

Common Mistakes

[!NOTE] S3 is the cheapest hard drive on earth. Architect around its strengths (immutable, durable, KV) and don't fight its weaknesses (latency, mutations).

Interview Follow-ups

• "How would you build a CDN-fronted bucket for video?" — CloudFront in front; signed URLs; bucket private; signed Cookie for whole-folder access.
• "How to manage millions of small files?" — Pack into parquet/tar; store index in DB; reduces S3 request cost dramatically.
• "S3 vs DynamoDB?" — S3 = large blobs, infrequent access pattern, eventual-now-strong consistency. DynamoDB = small structured records, milliseconds, transactions.

Q: Time-series databases — what makes them different?

Answer:

A time-series DB (TSDB) is optimized for append-only, timestamp-indexed data with high write volume and range-scan reads. Trying to do this in a normal RDBMS is a path to pain.

What Makes Data "Time-Series"

• Each record has a timestamp.
• New data is mostly appended at "now."
• Updates are rare; deletes are bulk (retention).
• Queries scan ranges by time: "last 5 min," "yesterday."
• Aggregations dominate: rate, sum, p99 over windows.

Use cases: monitoring metrics, IoT sensor data, financial ticks, user activity events.

Popular TSDBs

Optimizations

Append-only writes: no random writes; sequential disk patterns; high throughput.

Compression: Gorilla algo (timestamps as delta-of-delta, values as XOR diff) → ~1.4 bytes/sample.

Time-partitioned storage: each chunk covers a time window; old chunks dropped wholesale.

Indexes by (series, timestamp): range queries find the chunk + scan.

Downsampling: roll up raw → 1-min → 5-min → 1-hour at increasing retention.

Schema: Series

series_id = hash(metric_name + label_set)
data = [(ts, value), (ts, value), ...]

A series is a unique combination of labels. Each new label combination creates a new series.

Cardinality = total unique series. Bad labels (user_id, request_id) explode cardinality → OOM.

Querying

Prometheus PromQL example:

sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)

InfluxDB Flux:

from(bucket: "metrics")
  |> range(start: -1h)
  |> filter(fn: (r) => r._measurement == "cpu" and r.host == "h1")
  |> aggregateWindow(every: 1m, fn: mean)

TimescaleDB SQL:

SELECT time_bucket('1 minute', time) AS bucket, avg(value)
FROM cpu_usage
WHERE host = 'h1' AND time > now() - INTERVAL '1 hour'
GROUP BY bucket
ORDER BY bucket;

Retention

Time-based retention is built in:

keep raw for 7 days
keep 1-min downsampled for 30 days
keep 1-hour downsampled for 1 year

Dropping a time window = dropping a chunk file. Trivial.

vs Generic RDBMS

If you try Postgres for metrics at scale:

• 100k inserts/sec hits write throughput.
• Indexes bloat from constant inserts.
• Vacuum overhead.
• Time-range queries do sequential scan on huge tables.

TimescaleDB sidesteps this by chunking (auto-partitioning by time) on top of Postgres + columnar compression. Best of both: SQL + TSDB performance.

vs OLAP (ClickHouse, BigQuery)

OLAP DBs are great for time-series too. ClickHouse, Druid, Pinot:

• Columnar storage.
• Excellent aggregations.
• Higher query latency than dedicated TSDBs but more flexible (joins, distinct counts).

For metrics: TSDB. For events with rich querying: OLAP. The line blurs.

Cardinality Management

Sources of explosion:

• Per-user labels.
• Per-request IDs.
• Per-pod UUIDs in K8s.
• Free-form text labels.

Rule of thumb:

• Labels: < 10 values per label, < 10 labels.
• Cardinality budget per metric: ~10,000 series.
• High-cardinality goes to logs/events, not metrics.

Scale

Million-series TSDBs commonplace.
Tens-of-millions with sharded/clustered solutions.
Hundreds-of-millions: managed (Mimir, VictoriaMetrics cluster, M3DB).

Single node holds ~1M series, 100k samples/sec.

Multi-Tenant / Federated

For organization-wide metrics:

• Per-team / per-cluster TSDB.
• Federation: central queries fan out to clusters.
• Or: managed multi-tenant (Cortex, Mimir, Datadog).

Common Mistakes

[!NOTE] Time-series databases trade flexibility for compression + write performance. They're not "a SQL database with a timestamp column" — they're a specialized engine and need to be treated as such.

Interview Follow-ups

• "When would you NOT use a TSDB?" — Ad-hoc analytics with complex joins; rich relational queries — use a warehouse instead.
• "How would you store both metrics and traces?" — Metrics in TSDB; traces in dedicated trace store (Tempo, Jaeger); logs in Loki/Elastic. Three pillars, three stores.
• "What's downsampling, and what does it cost?" — Aggregate raw points into coarser buckets (1m → 5m → 1h) over time. Cost: small CPU to compute; you lose precision but save 12–60× storage.

Q: Cache strategies — cache-aside, write-through, write-back, write-around, read-through.

Answer:

A cache speeds up reads (sometimes writes) by sitting in front of a slower store. The five canonical strategies differ in who writes to the cache vs DB, and when. Pick by failure mode you can tolerate.

The Five Strategies

Cache-Aside (Most Common)

def get(key):
    val = cache.get(key)
    if val is None:
        val = db.get(key)
        cache.set(key, val, ttl=600)
    return val

def update(key, val):
    db.put(key, val)
    cache.delete(key)         # invalidate; don't update

Properties:

• Simple, app-controlled.
• Cache failures degrade gracefully (DB serves).
• Stale window between DB write and cache invalidate.
• Cache misses go to DB — stampede risk under burst load.

Why delete instead of set on writes: if two writes race, deletes are commutative; set can install a stale value.

Read-Through

Cache exposes the same API as the DB; the cache itself does the DB lookup on miss.

get(key):
  cache layer:
    val = local.get(key)
    if val is None:
        val = db.get(key)
        local.set(key, val, ttl)
    return val

Centralizes cache logic, simplifies the app. Most managed caches (AWS DAX) work this way.

Write-Through

Writes go to cache, cache writes DB synchronously.

write(key, val):
  cache.set(key, val)
  db.put(key, val)        # sync
  return success

Pros: cache always in sync. Cons: writes are slower (cache + DB latency). Cache must be in front of every write path — no out-of-band DB writes allowed (or you'll have drift).

Write-Back / Write-Behind

Writes to cache only; cache flushes to DB asynchronously (batched).

write(key, val):
  cache.set(key, val)
  schedule async DB write
  return success            ← fast

Pros: ultra-fast writes, batching for DB efficiency. Cons: data loss if cache dies before flush. Suitable for metrics, counters, analytics. Not for financial data.

Write-Around

Writes bypass cache, only DB is touched. Cache is filled on subsequent reads.

write(key, val):
  db.put(key, val)
  # do NOT write cache

read(key):
  val = cache.get(key)
  if val is None:
      val = db.get(key)
      cache.set(key, val)

Useful for write-heavy data that's rarely re-read (logs, telemetry).

Choosing

TTL — the Quiet Hero

Every cache entry should have a TTL. Reasons:

• Limits stale-data damage from bugs in invalidation.
• Bounds cache size organically.
• Keeps cache "honest" — if a key is wrong, it self-heals in TTL.

Typical TTLs:

• User profile: 5–60 min.
• Static reference data: hours/days.
• Anything counters or timeseries: seconds.

Negative Caching

Cache "no, this doesn't exist" responses too. Otherwise a flood of requests for nonexistent keys (e.g., bots probing) hits the DB.

val = db.get(key)
if val is None:
    cache.set(key, NULL_SENTINEL, ttl=60)    # short TTL

Refresh-Ahead

Refresh popular keys before they expire to avoid the miss stampede:

on read:
   val, ttl_remaining = cache.get_with_ttl(key)
   if ttl_remaining < 10% of original:
       async_refresh(key)
   return val

The user serving the request never waits on the refresh.

Multi-Tier Caches

Browser cache  (HTTP cache headers)
   ▼
CDN            (edge, geo)
   ▼
App-server local cache  (in-process LRU)
   ▼
Distributed cache  (Redis cluster)
   ▼
DB

Each layer absorbs progressively more traffic. p99 latency is set by the cheapest cache that's still warm enough.

Cache Hit Ratio

Most important cache metric.

hit_rate = hits / (hits + misses)

Healthy: > 80%. Excellent: > 95%. Below 50% — the cache is barely earning its keep; revisit keys, TTLs, capacity.

Common Mistakes

[!NOTE] The first rule of caching: invalidation is the hardest problem. The second rule: TTL is your friend — design assuming invalidation will sometimes fail.

Interview Follow-ups

• "How would you achieve strong consistency between cache and DB?" — Hard. Options: write-through with cache as source of truth; or accept eventual + short TTLs; or use CDC to invalidate cache from DB changes.
• "What about cache stampede on cold start?" — See Cache Stampede: request coalescing, locks, probabilistic early refresh.
• "How do you decide what to cache?" — Hot-set analysis: log key accesses, find the Pareto 20%. Cache that; ignore the rest.

Q: Cache invalidation & eviction — the hard part of caching.

Answer:

"There are only two hard things in computer science: cache invalidation and naming things." — Phil Karlton. The invalidation problem is real because cache and source-of-truth can diverge, and detecting / repairing the drift is the bulk of the engineering.

Invalidation Approaches

1. TTL (time-based)

Every key expires after N seconds. Simplest, most reliable. The cache self-heals; stale data has a bounded lifetime.

cache.set(key, value, ttl=600)

Trade: stale data lives up to TTL.

Pick TTL by:

• How sensitive the data is to staleness.
• How much load the origin can take if all cache entries expire at once.

2. Write-through invalidation

On every write to source, also update / delete cache:

write(key, value):
  db.put(key, value)
  cache.delete(key)         ← delete, don't set

Why delete vs set: race conditions. Two concurrent writers can set stale value last. Deleting is commutative.

Trade: every write path must remember to invalidate. Easy to miss in heterogeneous code.

3. Event-driven (CDC) invalidation

The DB streams its changes; a separate process invalidates cache entries based on the stream:

Postgres WAL → Debezium → Kafka → cache invalidator

Decouples invalidation logic from write paths. Works even if a write bypassed your app.

4. Versioning / tagging

Each cached entry includes a version or set of tags. Invalidation = bump version / purge tag.

cache.set("user:42:v3", value)            # version in key
cache.set_with_tags(key, value, tags=["user:42"])

# Invalidate:
cache.invalidate_tag("user:42")

Fastly / Cloudflare support tag-based purge for CDN.

Eviction Policies (When Cache Full)

Redis defaults: noeviction. Change via maxmemory-policy.

LRU Implementation

Doubly linked list + hash map:

• On access: move to head.
• On insert: add at head; if over capacity, remove tail.
• O(1) per operation.

LRU is the universal default. LFU often does better when access patterns have a long tail of "loved" keys, but is harder to implement (LFU classic doesn't decay; "windowed TinyLFU" fixes it).

Hot Key Eviction Problem

A single hot key + many less-popular keys + LRU = the hot key keeps the popular content in cache → less-popular evicted constantly → unstable behavior.

Fix: LFU or LRU + size-tier (keep large objects separate).

Negative Caching

Cache "no" answers too:

val = db.get(key)
if val is None:
    cache.set(key, NULL_SENTINEL, ttl=60)

Without it, repeated lookups for non-existent keys hit the DB. Bots and crawlers exploit this.

Short TTL — newly inserted keys should appear after a brief window.

Refresh-Ahead

Refresh popular keys before expiry:

on read:
    val, ttl_remaining = cache.get_with_ttl(key)
    if ttl_remaining < 0.1 * original_ttl:
        async_refresh(key)        # background
    return val

User never waits on refresh. Reduces stampede risk.

Stampede Protection

Hot key expires; many clients race to refill:

• Lock + fetch (first one wins; others wait).
• Probabilistic early expiration (each client treats TTL with jitter).
• Stale-while-revalidate (serve stale during async refresh).

See Cache Stampede.

Multi-Tier Invalidation

Browser cache (Cache-Control)
  ▼
CDN edge cache
  ▼
App-server local cache (Caffeine, lru-cache)
  ▼
Distributed cache (Redis)
  ▼
DB

Invalidating across all tiers is hard. Strategies:

• Tag-based purge at CDN (Fastly, Cloudflare).
• Short TTL at lower tiers.
• Bust local caches via pub/sub on invalidation event.

CDN-Specific Invalidation

purge by URL:    invalidate single URL
purge by tag:    invalidate group
soft purge:      serve stale until origin returns fresh

Purges are not instant — usually < 30s propagation.

Avoid: Cache Aside With Concurrent Updates

client A: write DB
client B: write DB
client A: invalidate cache
client B: invalidate cache
client A: SET cache = old A read  ← race

This is why we delete rather than set on writes. The next miss reads from DB and fills correctly.

Common Mistakes

Observability

• Hit ratio: hits / (hits + misses).
• Stale rate: how often was the cached value out of date when re-fetched?
• Eviction rate: indicates capacity vs working set.
• Invalidation events / sec: too many = bug in upstream.

[!NOTE] Treat invalidation as a separate, observable concern from the cache itself. The fact that "we have a cache" doesn't tell you whether it's correct — instrument what matters.

Interview Follow-ups

• "How do you keep cache consistent with DB?" — You don't keep them perfectly consistent. Pick: TTL (bounded staleness), write-through (synchronous cost), or CDC (eventual). All have trade-offs.
• "What's the right TTL?" — Long enough that the cache hits; short enough that stale damage is bounded. Measure both.
• "How do you cache personalized data?" — Per-user key prefix; smaller hit ratios but precise. Or cache fragments and assemble at request time.

Q: Cache stampede / thundering herd — what causes it and how to fix it.

Answer:

A cache stampede happens when a hot key expires (or a cache restarts) and many concurrent requests stampede the origin, often crushing the database. Easy to overlook; can take down production.

The Failure

t=0:    100k QPS hits cache; 99% hits.
t=10:   hot key expires.
t=10s:  100k requests all miss; all hit DB simultaneously.
t=10s:  DB collapses under the load it normally never sees.

The hotter the key, the worse the stampede.

Causes

1. Hot key expires: TTL hits; everyone races to refill.
2. Cache restart: Redis crash + reboot loses all keys.
3. Cache deploy / scaling event: new nodes start cold.
4. Mass invalidation: a tag purge nukes thousands of keys at once.

Mitigations

1. Locking / Request Coalescing

Only one request fetches; others wait:

def get(key):
    val = cache.get(key)
    if val is not None:
        return val

    if cache.setnx("lock:" + key, "1", ttl=10):       # try acquire
        try:
            val = db.get(key)
            cache.set(key, val, ttl=600)
            return val
        finally:
            cache.delete("lock:" + key)
    else:
        # Someone else is refreshing — wait briefly
        time.sleep(0.1)
        return get(key)        # retry

The first miss takes the lock; the rest poll. DB sees one request.

Variants:

• Java: Caffeine's LoadingCache does this in-process.
• Go: singleflight.
• Distributed: Redis SETNX or Redlock.

2. Stale-While-Revalidate

Serve stale during async refresh:

val, age = cache.get_with_age(key)
if age > TTL:                       # stale
    async_refresh(key)               # background
return val                           # serve stale anyway

The user sees no latency spike. Origin sees one refresh, not many.

Browser / CDN support this via Cache-Control: stale-while-revalidate=....

3. Probabilistic Early Expiration

Each client treats TTL as random within a window:

effective_ttl = ttl × (1 + random()×0.1)

Or smarter: XFetch algorithm — refresh probability rises as TTL nears:

on read:
   if random() < (now - expiry) / refresh_window:
       refresh()
   return cached_value

Spreads refresh across clients; only one wins the race naturally.

4. Permanent Cache + Background Refresh

Don't expire; refresh periodically. Cache is always populated; just possibly stale.

on read:        return cache.get(key)
every 60s:      cache.set(key, db.get(key))

Works for read-mostly data with bounded staleness budget. No stampede ever, because no expiry.

5. Pre-Warming on Deploy

Before promoting a new cache node, pre-populate it from the old:

new node joins cluster
   → drain replica or fetch from sibling
   → mark ready when caught up

Avoids cold-start stampede.

6. Hot Key Replication

For a single key with massive traffic:

• Replicate the key to all cache nodes (read-anywhere).
• Or process-local L1 cache for that key only.
• Reduces request to a single shard.

Stampede Scenarios by Scale

Detection

Metrics:

• Backend QPS spike correlated with cache miss spike.
• p99 latency briefly degraded.
• Lock contention metrics (if you're using locks).

Alert on:

• cache_misses_per_sec spike > N× baseline.
• DB request count anomaly.

Common Mistakes

[!NOTE] Cache stampedes are silent killers. They don't show up under steady state — only when the cache is "doing its job" most of the time and a hot key happens to expire. Test stampede behavior explicitly (drop a hot key, watch the origin).

Interview Follow-ups

• "What's the cheapest fix?" — TTL jitter + SWR. No new infra, minimal code.
• "How does Caffeine handle this in-process?" — LoadingCache with a CacheLoader; only one thread computes per key; others wait on the result.
• "How would you handle a cold-start stampede after Redis restart?" — Replay-from-replica + canary connection draining + connection-pool limit toward DB (force backpressure).

Q: Redis vs Memcached — picking a distributed cache.

Answer:

The two dominate distributed caching. Memcached is older, simpler, faster on pure key-value. Redis is richer, with data structures, persistence, pub/sub, scripting. Modern default: Redis unless you need pure speed and nothing else.

Memcached

Pure in-memory key-value store. Multi-threaded.

SET key value 0 60        # value, flags, expiry
GET key
DELETE key

• One data type: opaque blob.
• LRU eviction.
• No persistence.
• Multi-threaded (scales on a single box with many cores).
• Sharding: client-side consistent hashing.
• Operations strictly per-key; no multi-key transactions.

Strengths:

• Simplest semantics.
• Fastest pure get/set.
• Predictable memory model.

Redis

In-memory data-structure store. Single-threaded core + I/O threading.

Data types:

• Strings (incl. counters).
• Lists.
• Sets, Sorted Sets.
• Hashes.
• Streams.
• Bitmaps, HyperLogLog, Geo.

Operations:

• Transactions (MULTI / EXEC).
• Lua scripting (atomic).
• Pub/Sub.
• Streams (consumer groups, like a mini Kafka).
• Persistence (RDB snapshots + AOF append-only file).
• Replication + Cluster mode.

Strengths:

• Much more than a cache (queues, leaderboards, locks, rate limiters).
• Persistence option.
• Richer client libraries.

When To Pick Which

For 90% of new projects: Redis. Memcached's niche is rare these days.

Persistence (Redis)

• RDB: snapshot to disk every N seconds. Fast restore; some loss.
• AOF: append every write to a log. Slower; minimal loss (configurable fsync).
• Hybrid: AOF for durability + RDB for fast restore.

For pure cache: persistence off; treat as ephemeral. For session store / queue / counter: AOF with everysec fsync — good balance.

Replication

master → slave (or in modern terms, primary → replica)

Async by default. Replica serves reads (optionally); failover if primary dies.

Redis Sentinel monitors and orchestrates failover.

Redis Cluster

16,384 hash slots assigned across nodes. CRC16(key) → slot → node.

3-master, 3-replica cluster:
  M1 (slots 0-5460)        ← R1
  M2 (slots 5461-10922)    ← R2
  M3 (slots 10923-16383)   ← R3

Client gets MOVED redirect if it lands on wrong node. Multi-key operations require keys in same slot (use {} hash tags: user:{42}:profile and user:{42}:settings share a slot).

Memcached Sharding

Pure client-side. The client library hashes the key and picks the node:

node = consistent_hash(key) → server in ring

No coordinated state. Adding a node moves ~1/N of keys.

Throughput Benchmarks

Single node:

• Memcached: ~1.5M ops/sec/core (multi-threaded).
• Redis: ~100k–200k ops/sec single-threaded; more with pipelining + io_uring.

Multi-core scaling:

• Memcached scales linearly across cores in one process.
• Redis traditionally requires multiple instances (or use KeyDB, Dragonfly for multi-core Redis-protocol-compatible).

Pipelining

Both support pipelining: send N requests in one network round trip.

pipeline.set(a, 1).set(b, 2).get(c).execute()

10× throughput improvement under high QPS. Critical for production tuning.

Eviction

• Memcached: LRU per slab class.
• Redis: configurable (maxmemory-policy):
  • noeviction: reject writes when full.
  • allkeys-lru, allkeys-lfu, allkeys-random.
  • volatile-lru, etc. (only keys with TTL).

For pure cache: allkeys-lru typical. For mixed workload (cache + counters that must not be evicted): set TTL only on cacheable keys and use volatile-lru.

Operational Concerns

• Memory fragmentation: monitor used_memory_rss / used_memory. Restart if high.
• Persistence cost: AOF fsync can stall the event loop.
• Big keys: a 1 GB string is fine to fetch once; killing if accidentally enumerated. Avoid.
• SLOWLOG: identify slow commands.
• Cluster slot reshard during traffic: smooth via streaming; brief blips.

Common Mistakes

Alternatives

• KeyDB: multi-threaded Redis fork.
• Dragonfly: modern Redis-compatible with much higher throughput.
• Hazelcast / Ignite / Aerospike: enterprise in-memory data grids.
• Cloud-managed: ElastiCache, MemoryDB (durable Redis), Memorystore.

For pure cache at scale: Memcached or Dragonfly. For everything else: Redis.

[!NOTE] The choice often comes down to ecosystem comfort and team familiarity. Both work. Pick the one that matches your durability + data-structure needs.

Interview Follow-ups

• "How do you ensure consistency between Redis cache and DB?" — Cache-aside pattern with delete-on-write + TTL. Never set the cache on write (race).
• "How would you build a distributed lock in Redis?" — SETNX with TTL; Redlock for multi-node Redis (controversial — see Martin Kleppmann's critique). For most cases, single-instance SETNX is fine.
• "What about MemoryDB?" — AWS's durable Redis (multi-AZ persistence). For workloads that need Redis API + DB-grade durability.

Q: Queue vs Pub/Sub vs Log — which messaging model when?

Answer:

"Messaging" is three distinct patterns with very different semantics:

• Queue: one producer, one consumer per message. Work distribution.
• Pub/Sub: one producer, many consumers per message. Fan-out.
• Log: ordered, replayable record. Pub/sub + history.

Get the model right; the broker choice (Rabbit, SQS, Kafka) follows.

Queue

producer ──► [ msg1, msg2, msg3 ] ──► consumer1 (gets msg1)
                                    ──► consumer2 (gets msg2)
                                    ──► consumer3 (gets msg3)

Each message delivered to exactly one consumer. Workers compete; throughput scales with workers.

Used for: background jobs, email sending, image processing, async workflows.

Brokers: RabbitMQ, AWS SQS, Google Pub/Sub (pull mode), Beanstalkd, NATS JetStream.

Semantics:

• At-least-once is the typical default; consumer ACKs each message after processing.
• Visibility timeout: message hidden after delivery; redelivered if not ACKed in window.
• DLQ: after N failures, route to dead-letter queue for inspection.

Pub/Sub

producer ──► topic ──► subscriber A (sees all messages)
                   ──► subscriber B (sees all messages)
                   ──► subscriber C (sees all messages)

Each message delivered to every subscriber. Subscribers usually independent — A processing doesn't affect B.

Used for: event notifications, cache invalidation, real-time updates.

Brokers: Redis Pub/Sub (ephemeral), Google Pub/Sub (push), MQTT (IoT), Kafka (topics with consumer groups).

Semantics:

• Most pub/sub is at-least-once.
• If a subscriber is offline, may miss messages (depending on broker).
• Filtering: SNS message filters, AMQP routing keys.

Log

producer ──► append-only log ──► consumer A reads from offset 0
                              ──► consumer B reads from offset 0
                              ──► consumer C reads from offset 1000

   [m1][m2][m3][m4][m5][m6][m7]...   ← persistent, ordered

A durable, ordered record. Consumers maintain their own offset; can replay from any point. Multiple independent consumer groups read the same log.

Used for: event sourcing, stream processing, audit trails, CDC, data pipelines.

Brokers: Apache Kafka, AWS Kinesis, Apache Pulsar, Redpanda.

Semantics:

• Ordering preserved per partition.
• Retention by time or size, not by ACK.
• Replay: a new consumer can start from offset 0 and reprocess history.

The Three Compared

Picking the Right One

Decision tree:

Need persistent history? ─── yes ──► Log (Kafka)
              │
              no
              │
Same message to many independent systems? ─── yes ──► Pub/Sub
              │
              no
              │
Multiple workers competing? ─── yes ──► Queue

Combinations are common:

• Producer publishes to Kafka (log) → many consumer groups read.
• Each consumer group might internally distribute work via SQS (queue) for fan-out within the service.

Delivery Guarantees

Most brokers do at-least-once. Exactly-once is achieved via consumer-side idempotency keys.

See Delivery Semantics.

Ordering

Strict global ordering = single producer + single consumer. Doesn't scale.

Practical: ordering per key.

• Kafka: same key → same partition → ordered.
• SQS FIFO: messages share a MessageGroupId → ordered within group.
• Pulsar: keyed_shared subscriptions.

Cross-key ordering: not guaranteed; design your business logic to be commutative or sequenced by app.

Push vs Pull

Pull-based usually handles backpressure better — consumer reads at its own pace.

Backpressure

What happens when the consumer can't keep up?

See Backpressure.

When Not to Use a Broker

• Synchronous request/response with low fanout — just call the service.
• Workflows requiring strict ordering across many entities — durable workflow engine (Temporal, Camunda).
• Sub-millisecond replies — broker hops add latency.

Common Mistakes

[!NOTE] The hardest model to get right is "exactly-once event processing." Stop fighting it: use at-least-once + idempotent consumers + Idempotency-Key. That's how real systems do it.

Interview Follow-ups

• "How do you ensure ordering across services?" — Per-key ordering via partition/group; for cross-key, sequence via a single owner (saga orchestrator) or accept eventual.
• "What's the difference between Kafka topic and SQS queue?" — Topic = log with partitions, consumer groups, retention. Queue = transient buffer, one delivery per message, no replay.
• "How would you do exactly-once delivery?" — You don't. Use at-least-once + dedup by event_id on the consumer side (DB unique constraint or seen-set).

Q: At-most-once, at-least-once, exactly-once — what they mean and how to achieve them.

Answer:

Delivery semantics describe what happens when messages are sent across a network that may fail. They're not properties of brokers; they're properties of producer + broker + consumer + dedup working together.

The Three Semantics

At-most-once

• Send and forget.
• Message may be lost.
• Never duplicated.
• Fastest.

At-least-once

• Retry until acknowledged.
• May duplicate.
• Never lost.
• Most common default.

Exactly-once

• Single observable effect.
• Hardest; requires producer + consumer + storage coordination.

Achieving Each

At-most-once:

producer.send(msg)
# no retry on failure
# no ack required

OK for: telemetry where freshness > completeness; UDP-like protocols.

At-least-once:

producer.send(msg)
on ack timeout: retry with backoff

Plus:

• Broker durability (replication, fsync).
• Consumer commits offset AFTER processing.
• Idempotent processing.

This is what 99% of production systems use.

Exactly-once:

Two flavors:

• Exactly-once delivery: impossible over an unreliable network (Two Generals Problem).
• Exactly-once effect (idempotent): achievable; producer can duplicate, but consumer ensures the work happens once.

How to Make Effects Exactly-Once

1. Idempotent consumer.

Consumer dedups by event ID before applying:

INSERT INTO processed_events (event_id) VALUES (?) ON CONFLICT DO NOTHING;
-- if conflict: skip
-- else: do the work

event_id from producer, immutable.

2. Transactional consumer.

Wrap "do work + commit offset" in one transaction:

begin
  apply event (idempotent OR uses event ID)
  commit consumer offset to same store
commit

Kafka EOS_V2: producer transactions + consumer reads read_committed to skip aborted records.

3. Outbox + inbox pattern.

Producer atomically writes outbox + business data. Consumer atomically writes inbox + applies effect.

See Transactional Outbox.

Kafka's Exactly-Once Story

• Idempotent producer (enable.idempotence=true): broker dedups by (producer_id, seq). Eliminates dupes within one producer session.
• Transactions: producer writes atomically to multiple topics + commits consumer offsets in one tx. Consumer with isolation.level=read_committed skips aborted records.
• End-to-end: app must use Kafka transactions correctly and read with read_committed.

Cost: ~20–40% throughput hit; small latency increase. Worth it for billing pipelines and similar.

Common Cases

Order placement webhook:

• At-least-once + idempotency key.
• Processor dedups by key.
• "Exactly-once effect."

Email sending:

• At-least-once + dedup by dedup_key.
• Bouncing duplicate sends → just don't send the email again.

Counters / analytics:

• At-least-once + dedup by event_id at the aggregator.
• Or accept small overcount for higher throughput.

Financial ledger:

• At-least-once + idempotency key.
• Append-only ledger; duplicate inserts rejected by unique constraint.
• Reconciliation pass catches anything that slipped through.

Where Each Layer Can Fail

producer ─send→ broker ─store→ consumer ─process→ effect ─commit→

A failure between any two stages requires the next attempt to be idempotent.

Common Mistakes

[!NOTE] Stop asking for exactly-once delivery. Ask for exactly-once effects — that's achievable with idempotent operations and a dedup table. The semantics of the wire are at-least-once; the semantics of your business logic must be at-most-once on top.

Interview Follow-ups

• "How does Kafka achieve exactly-once across two topics?" — Transactional producer wraps writes + offset commit; consumer reads read_committed. Atomic from the consumer's POV.
• "What if the dedup store is down?" — Fail-closed (reject the message) or fail-open (allow duplicate). Pick per business cost.
• "Why is exactly-once delivery impossible?" — Two Generals Problem: in an unreliable network, you can never know with certainty that your peer received the message. Retries are necessary; therefore duplicates are unavoidable; therefore consumer dedup is mandatory.

Q: Backpressure — what is it, and how do you implement it?

Answer:

Backpressure is the feedback signal from a slow consumer telling its producer to slow down. Without it, producers overwhelm consumers, queues balloon, latency spikes, and the system collapses. Every streaming and async system needs an explicit backpressure story.

The Failure Mode

producer → unbounded queue → consumer (slow)

Queue grows: 1k, 10k, 100k, 1M messages.
Memory pressure builds. GC churns.
Latency for old messages goes from ms to minutes.
Eventually: OOM, or consumer can never catch up.

The Fix: Apply Backpressure

The producer must respond to the consumer's pace. Three flavors:

1. Blocking. Producer's send() blocks until consumer has capacity.

bounded queue (capacity=1000)
   send() blocks if full
   recv() unblocks send when capacity returns

Simple. Trade: blocked producer threads.

2. Lossy. Drop messages when full.

try_send(msg):
  if queue.size < capacity:
    queue.add(msg)
  else:
    drop or sample

Used for telemetry, analytics where freshness > completeness.

3. Reactive (pull-based). Consumer signals demand; producer sends only what's requested.

consumer.request(10)   → producer sends 10 items
consumer.request(5)    → producer sends 5 more

Used by Reactive Streams (Java/JVM), RxJS, Akka Streams.

Where Backpressure Lives Per Tech

Application-Level Patterns

1. Bounded buffer.

queue.put(msg)        # blocks if full

Java's BlockingQueue capacity-bounded. Producer naturally throttles.

2. Semaphore for concurrency control.

sem = Semaphore(100)
for task in tasks:
    sem.acquire()
    async_process(task).then(sem.release)

Caps in-flight work; doesn't let it spike unboundedly.

3. Token bucket producer.

Producer holds a bucket of tokens, replenished at consumer's confirmed pace. Sends only with a token.

4. Adaptive rate limit.

Producer measures consumer's response latency; if latency rises, reduce send rate. AIMD (Additive Increase, Multiplicative Decrease) — same as TCP congestion control.

When Backpressure Isn't Enough — Shed Load

If you can't slow down, drop messages. Patterns:

• Tail drop: drop new messages when full.
• Head drop: drop oldest (newest data more valuable, e.g., telemetry).
• Priority drop: drop low-priority first.
• Random sampling: drop 1 in N to maintain a representative sample.

Better than collapsing. Document what's dropped.

Async / Reactive Backpressure

Reactive Streams spec defines a Publisher-Subscriber protocol with request(n):

Flux.range(1, 1000)
    .onBackpressureBuffer(100)       // buffer up to 100
    .onBackpressureDrop(msg -> log("dropped: " + msg))
    .subscribe(subscriber);

Operators like buffer, drop, latest define overflow strategy.

Buffer Sizing

buffer_size = expected_lag × throughput

If consumer might lag by 1s and you process 10k/sec, buffer of 10k handles 1s spike.

Larger buffers smooth bursts; smaller buffers reveal problems faster. Tune for your SLA.

Backpressure vs Throttling

• Backpressure: feedback from consumer to producer.
• Throttling / rate limiting: producer caps itself (or is capped by gateway).

Both reduce flow; backpressure is reactive (responds to actual capacity); throttling is proactive (predefined limit).

Multi-Layer Backpressure

client ─── HTTP/2 flow control ──► gateway
                                     │
                                     │ bounded channel
                                     ▼
                                  service
                                     │
                                     │ async submit with semaphore
                                     ▼
                                  worker pool
                                     │
                                     │ bounded queue
                                     ▼
                                  downstream

Each layer enforces capacity. The slowest layer signals back through every prior layer.

Common Mistakes

Observability

• queue_size (gauge).
• queue_oldest_age (oldest pending item's age).
• producer_blocked_total (counter).
• dropped_messages_total (counter, with reason label).

Alerting on these surfaces backpressure before users feel it.

[!NOTE] Backpressure is not optional. Every async boundary in a system needs an explicit policy: bound the buffer, define overflow, surface the metric. "Just make the queue bigger" is how systems die.

Interview Follow-ups

• "How does Kafka apply backpressure?" — Consumer pulls. If consumer can't keep up, lag grows; producer keeps producing into the log. Backpressure to producer happens only if log retention is exceeded (very long lag) — by then it's a real outage.
• "What about WebSocket backpressure?" — Per-connection send buffer; if app can't drain client's slow socket, write to socket blocks; either drop or disconnect slow client.
• "How does AIMD relate?" — TCP congestion control mimics what application backpressure should do — slowly raise rate, halve it on signs of trouble.

Q: Consensus — Raft and Paxos. What do they solve, how do they work?

Answer:

Consensus protocols let a cluster of nodes agree on a value despite failures and message delays. They're the foundation of every distributed system that claims correctness: etcd, Consul, ZooKeeper, Spanner, CockroachDB, Kafka KRaft.

The Problem

Multiple servers must agree on:

• Who is the leader.
• The order of operations in a replicated log.
• Membership of the cluster.

Despite:

• Crashes (fail-stop).
• Network partitions.
• Message reordering, delay.

But not despite:

• Byzantine failures (servers lying) — outside scope; need different protocols.

Properties

A correct consensus protocol guarantees:

• Safety: agree on the same value (no two correct nodes decide differently).
• Liveness: eventually decide a value (assuming a majority is up).

Famous result: FLP impossibility — no deterministic protocol guarantees both safety and liveness in a fully async network with even one crash. In practice, we use timeouts and accept brief liveness violations during failures.

Raft (Understandable)

Raft splits consensus into three sub-problems:

1. Leader election.
2. Log replication.
3. Safety (rules ensure only safe logs win elections).

Election

Each node is Follower, Candidate, or Leader.

Follower: hears heartbeats from leader; if timeout, become Candidate.

Candidate:
  ++current_term
  vote for self
  request votes from peers

Receive majority of votes? → become Leader.
Receive heartbeat with higher term? → become Follower.
Election timeout? → start new election.

Election timeout randomized to prevent split votes (e.g., 150–300 ms).

Log Replication

Leader receives client requests, appends to its log, replicates to followers.

client → leader.append(cmd)
leader: append to its log at index i
leader: send AppendEntries to all followers
followers: write to log; reply OK
leader: when majority confirmed → mark index i committed
leader: apply to state machine; respond to client
leader: tell followers of new commit index on next heartbeat

Once committed, the entry is durable — survives leader failure.

Safety

Rules ensure no leader is elected without all committed entries:

• Vote denied to candidate whose log is shorter / older.
• Leader never overwrites committed entries.
• Log matching property: same index + term → same prefix.

Paxos

Older, more general, harder to understand. Multi-Paxos for replicated logs.

Two phases per decision:

1. Prepare: proposer picks a number n, asks acceptors to promise not to accept lower n.
2. Accept: with a majority of promises, proposer asks acceptors to accept its value.

Leader-based optimizations (Multi-Paxos) skip Phase 1 once a leader is stable — looks similar to Raft in practice.

Paxos paper is notoriously dense. Raft was designed (and named) for understandability.

Raft vs Paxos

Practical guidance: use Raft unless you have a specific reason for Paxos. Most modern systems do.

Where Consensus Is Used

Quorum

Most operations need a majority of nodes (floor(N/2) + 1).

• 3 nodes → 2 quorum → tolerate 1 failure.
• 5 nodes → 3 quorum → tolerate 2 failures.
• 7 nodes → 4 quorum → tolerate 3 failures.

Larger N tolerates more failures but writes slow (must wait for majority).

Best practice: 3 or 5. Odd numbers avoid split-brain ties.

Failure Modes

Performance

• Write latency = 1 RTT to quorum (best case).
• Read latency: from leader (cheapest); from follower (eventual).
• Throughput: bounded by leader's I/O.

Scale beyond one Raft group: shard the keyspace, one Raft group per shard (Spanner, CockroachDB do this).

Pitfalls

• Split brain after partition heals: the protocol prevents it, but only if implemented correctly.
• Slow consensus over WAN: round-trip dominates; consider regional clusters.
• Disk fsync stalls: AppendEntries requires fsync; cheap drives kill throughput.
• Membership changes: must use joint consensus or single-node-at-a-time, not raw replacement.

Common Mistakes

[!NOTE] Consensus is the gold-plated screw of distributed systems. Use it for the small piece of state that must be linearizable across nodes. Build the rest on cheaper primitives (gossip, eventual consistency).

Interview Follow-ups

• "How does Raft handle leader failure?" — Election timeout triggers; candidates emerge; majority elects new leader; minimum disruption to clients (retries on new leader).
• "What is joint consensus?" — Raft's safe membership-change protocol: both old and new configurations active during transition.
• "Difference between Raft and Multi-Paxos?" — In practice, similar performance. Raft has stricter leader behavior; Paxos allows proposals from any node. Raft easier to implement correctly.

Q: How do you elect a leader in a distributed system?

Answer:

Many systems need exactly one node to perform a role (write coordinator, scheduler, cron). Leader election picks that one node and re-picks when it dies. The wrong implementation produces split-brain; the right one uses a consensus-backed lease.

Why You Need a Leader

• Single writer (avoids conflict resolution).
• Coordinator for distributed work (sharding, locking).
• Reduce thundering herd (one node decides, others follow).

Examples: Kafka controller, Raft leader, Kubernetes controller-manager, scheduled-job runner.

Naive Approaches (Don't)

Lowest IP wins. Works until two nodes disagree about who's "up."

Random: two could pick simultaneously.

Acquire a row lock in DB: lock holder is leader. Sort of works; DB outage = no leader.

These have split-brain risk: two leaders run simultaneously, both perform "exclusive" work, data corrupts.

The Right Pattern: Lease + Consensus

Coordinator (etcd / ZooKeeper / Consul)
   acquire-lease(key="leader", ttl=30s)
      if no one holds it: grant
      else: deny

Leader periodically renews the lease (e.g., every 10s, before TTL expiry).
If leader fails to renew → lease expires → another candidate acquires.

The coordinator runs Raft/Paxos internally; the lease grant is linearizable.

etcd Example

lease = etcd.lease(ttl=30)
ok = etcd.put("leader", "node-1", lease=lease.id, prevExist=False)
if ok:
    print("I am leader")
    while running:
        lease.keepalive()
else:
    watch("leader")     # become leader when current expires

ZooKeeper Pattern

Ephemeral sequential nodes:

each candidate creates /election/node_xxxx (auto-numbered, ephemeral)
the one with the lowest number is the leader
candidates watch the node ahead of them
if that node dies (session closes → ephemeral disappears) → next one becomes leader

ZooKeeper sessions ensure failed nodes are reliably detected.

Kubernetes Lease Object

The standard way in K8s:

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: my-controller
spec:
  holderIdentity: pod-1
  leaseDurationSeconds: 15
  renewTime: "..."

Controllers compete by updating the Lease object. K8s API server (backed by etcd) gives linearizable updates.

Built into client-go and operator SDKs.

Common Mistakes

Split-brain via short network blip.

Leader's link blips, lease expires, follower takes over. Original leader's network recovers — it still thinks it's leader. Now two leaders.

Mitigation: leader checks its lease before each operation; abdicates if lease has expired or been taken. Fencing tokens (see below) make this bulletproof.

Fencing Tokens

Each lease grant comes with a monotonic counter. Downstream systems reject writes from older fence tokens:

acquire-lease → grant with fence=42
   → write to DB with fence=42

later:
acquire-lease → grant with fence=43
   → write to DB with fence=43

old leader (still thinks it's leader) tries to write with fence=42
   → DB rejects (fence < current)

Required for correctness when downstream systems matter (DBs, queues).

Implemented in: etcd revision numbers, ZooKeeper zxid, Kafka epochs.

When Leader Election Fails

• No consensus possible (majority of coordinator nodes down): no leader; system degrades.
• Repeated elections (flapping): usually network instability; increase lease TTL.
• Two leaders simultaneously: bug. Add fencing tokens, verify lease before critical operations.

Performance

• Lease renewal cost: 1 round trip every N seconds (cheap).
• Election after failure: lease TTL + acquire time (~seconds).
• Acceptable for control-plane work; not for hot path.

Use Cases by Frequency

• High-frequency coordination (e.g., per-request leader): wrong tool; restructure.
• Per-job leader (cron, distributed scheduler): yes.
• Per-partition leader (Kafka): each partition has its own.
• Cluster-wide leader (k8s controller, Kafka controller): yes.

Trade-offs

Long TTL: faster failover, lower coordinator load, longer outages.

Short TTL: quicker failure detection, higher coordinator chatter, brief flap windows.

Typical: 15-30 second TTL with 5-second renewal.

Common Mistakes

[!NOTE] The lesson of every distributed-systems incident report: trust your lease + use fencing tokens. Without them, "we thought we had one leader" is the punchline.

Interview Follow-ups

• "How does Kafka elect the controller?" — In KRaft, Raft itself among the controller quorum. In ZK mode, candidates registered ephemeral znode; first wins; reelection on failure.
• "What's bully algorithm?" — Classic textbook leader election by node ID. Educational; not used in production (no fencing, sensitive to clock skew).
• "Why not just use a DB row lock?" — Works for simple cases; doesn't handle the DB's own failover, can't fence downstream operations.

Q: Distributed locks — how to build them, when to use them, when not to.

Answer:

A distributed lock makes one node "the only one doing X" across a cluster. It's tempting and dangerous in equal measure. Get it wrong and you have data corruption; get it right and you've added latency and a coordinator dependency.

When You Want a Distributed Lock

• Singleton workers (only one runs the migration).
• Per-resource serialization (only one process writes to file X).
• Coordinated leader work.

When You Don't

• Hot path (every request acquires lock → serialization bottleneck).
• Replacing transactions in your DB (use the DB's locking).
• Idempotency (use idempotency keys instead — no coordination needed).
• "Just in case" — every lock is a SPOF you didn't have before.

Implementation Options

1. Redis SETNX (single instance)

SET lock:key unique_value NX EX 30
   - NX: only if not exists
   - EX 30: TTL 30 seconds

Release:

-- Atomic Lua: delete only if I own it
if redis.call('GET', KEYS[1]) == ARGV[1] then
    return redis.call('DEL', KEYS[1])
end
return 0

The unique_value (request UUID) prevents Bob from deleting Alice's lock.

Properties:

• Single point of failure (Redis goes down → no lock).
• Lease-based via TTL — handles holder crash.
• Cheap, ~ms latency.

Best for: short-lived locks (< 30 s) where SPOF risk is acceptable.

2. Redlock (multi-instance Redis)

Acquire lock from majority of N independent Redis instances:

for each redis in [r1, r2, r3, r4, r5]:
    try acquire with TTL T
   
if majority acquired AND total time < T:
    lock held
else:
    release everywhere and retry

Goal: tolerate Redis instance failures.

Famous critique: Martin Kleppmann (2016) argues Redlock lacks correctness guarantees under clock skew / GC pauses. Antirez (Redis author) disagrees. For most use cases, Redlock is fine if you have fencing tokens downstream.

3. etcd / ZooKeeper / Consul

Backed by Raft. Linearizable lease management.

etcd:

lease = etcd.lease(ttl=30)
ok = etcd.put("lock:key", "owner", lease=lease.id, prevExist=False)
if ok:
    # I hold the lock
    do_work()
    lease.revoke()

Properties:

• Linearizable.
• Survives failures up to quorum.
• Higher latency than Redis (~10s of ms).
• Operates as both lease store + watcher.

Best for: correctness-critical locks where a few ms latency is acceptable.

4. Database lock

-- Postgres advisory lock
SELECT pg_try_advisory_lock(hash_key);
   -- ... do work ...
SELECT pg_advisory_unlock(hash_key);

-- Or for a row:
BEGIN;
SELECT * FROM resources WHERE id = ? FOR UPDATE;
   -- ... do work ...
COMMIT;

Properties:

• DB is already in the trust set.
• Limited by DB capacity.
• No TTL on advisory locks → holder crash holds lock until session ends.

The Lock Holder Dies Problem

Critical: what if the lock holder crashes mid-work?

• Without TTL: lock held forever.
• With TTL: lock expires, someone else takes over. But original holder might still be doing work (slow GC, paused process, network blip). Two holders. Bug.

Solution: fencing tokens.

Fencing Tokens

Lease grant includes a monotonically increasing token. Every downstream operation includes the token. Downstream rejects writes with stale tokens.

Alice acquires lock → fence=10
Alice's GC pauses 30s
Alice's lock expires
Bob acquires lock → fence=11
Bob writes to DB with fence=11

Alice resumes, tries to write with fence=10
DB sees fence < current → reject

Without fencing tokens, distributed locks can't be correct. Period.

Implementing: etcd revision, ZK zxid, Kafka epoch — all support fence-like semantics.

Locking Latency

• Redis single-instance: 1–2 ms.
• Redlock: 5–20 ms (depending on cluster geometry).
• etcd: 10–50 ms.
• DB advisory lock: depends on DB load.

Hot path that needs 1k+ QPS with a lock → wrong design. Restructure to avoid coordination.

Lock Granularity

• Coarse: one lock for all updates to a resource. Simple; reduces concurrency.
• Fine: one lock per row / key. More concurrent; more lock churn.

Match granularity to contention. Lock the smallest scope necessary.

Avoiding Locks Entirely

Better alternatives when applicable:

1. Optimistic concurrency (version columns).

UPDATE row SET ..., version = version + 1
WHERE id = ? AND version = ?

No lock; retry on mismatch.

2. Single-writer per partition (Kafka).

If one entity has one partition, only one consumer processes it. No lock needed.

3. Idempotency.

If retries are safe, you don't need to prevent them.

4. CRDTs.

Mergeable types eliminate conflict resolution.

Common Mistakes

[!NOTE] Distributed locks are a sharp tool. Two-line code with subtle correctness implications. Use them sparingly; prefer designs that don't need them.

Interview Follow-ups

• "How do you implement a lock with retry?" — acquire() retries on contention with backoff + jitter; give up after max attempts; alert if pattern persists.
• "What if a follower in etcd thinks it's leader?" — Raft ensures only the elected leader can perform consistent reads/writes; lease grants are linearizable.
• "Redlock or etcd for production?" — etcd if you need provable correctness; Redlock if you have fencing tokens and Redis is already in the stack.

Q: Two-phase commit (2PC) — and why you should avoid it.

Answer:

2PC is the classical protocol for atomic transactions across multiple participants. It works on paper, fails in production. Modern systems prefer Saga + idempotency over 2PC almost everywhere.

The Protocol

Coordinator + N participants.

Phase 1 (prepare):

coordinator → all participants: PREPARE
participants: do the work, lock resources, write to log
participants → coordinator: VOTE YES or NO

Phase 2 (commit/abort):

if all YES:
   coordinator → all: COMMIT
   participants: apply changes, release locks
else:
   coordinator → all: ABORT
   participants: roll back

Each participant is either committed or rolled back; never partial.

Why It Works (In Theory)

Each participant logs its decision before voting. If anyone crashes, on recovery they consult their log and the coordinator to learn the final outcome.

Why It's Bad (In Practice)

1. Blocking on coordinator failure.

If the coordinator dies after Phase 1 + some Phase 2 messages sent but not others, participants are left holding locks waiting for the coordinator to recover. Could be hours.

coordinator says PREPARE to A, B, C
all reply YES
coordinator crashes
A, B, C: locked, waiting for commit/abort

No reliable way to resolve without coordinator. Other transactions wait.

2. Holding locks across a network.

Participant locks rows during PREPARE. Lock duration = network round trip × 2. Cross-DC = hundreds of ms. Throughput plummets.

3. Doesn't scale.

Coordinator is a bottleneck; participants block on each other. Latency = max(slowest participant).

4. Doesn't handle some failures.

If a participant votes YES but crashes before receiving COMMIT, on recovery it must reconcile. Possible, but adds protocol complexity (3PC tried, mostly failed).

Where 2PC Lives Today

• XA transactions in JDBC + JMS: classic 2PC pattern. Performance and operational nightmare; rarely used.
• Internal distributed-DB protocols: Spanner uses 2PC under the hood across shards (with Paxos for durability). Works because Google built it carefully and locks are short.
• CockroachDB, YugabyteDB: similar Spanner-like distributed-SQL approach.

If you find yourself implementing 2PC in app code, stop.

Replacements

1. Saga.

Sequence of local transactions + compensating actions. Eventually consistent; no global lock. See Saga Pattern.

2. Transactional outbox.

Atomic write to DB + outbox in one local transaction. Downstream catches up async. See Transactional Outbox.

3. Distributed SQL.

If you really need ACID across data, use a system designed for it (Spanner, CockroachDB) and let it handle 2PC internally.

4. Single-writer per entity.

If only one node writes to a given entity, no coordination is needed.

Three-Phase Commit (3PC)

Adds a "PRE-COMMIT" phase between PREPARE and COMMIT. Reduces blocking risk; doesn't eliminate it in async networks. Theoretically interesting; not used in practice.

When 2PC Is Acceptable

• Small number of participants (2–3).
• Fast network (LAN, same DC).
• Short critical section.
• Failure tolerance acceptable (you're OK with manual recovery if coordinator dies).

Even then, the design conversation should justify why no alternative works.

Common Mistakes

[!NOTE] 2PC fails open in the wrong direction: when in doubt, participants block, not commit. That means any coordinator failure can cause an unbounded outage. Avoid it.

Interview Follow-ups

• "How does Spanner do distributed transactions?" — 2PC within Spanner, but each participant is itself a Paxos group; coordinator failure tolerated by majority. TrueTime narrows the lock window.
• "What's the Saga alternative?" — Sequence of local transactions; each step has a compensation. No global locks; eventual consistency.
• "What did Google's MegaStore / Spanner papers teach us?" — That 2PC + Paxos + clock synchronization + small batches can scale, with massive engineering effort. Most teams don't have that bandwidth.

Q: Distributed clocks — Lamport, vector, hybrid logical.

Answer:

Wall-clock time on different machines disagrees. Distributed systems use logical clocks to reason about event ordering without relying on synchronization. Three flavors: Lamport, vector, hybrid logical.

Why Wall Clocks Fail

Node A's clock: 12:00:00.500
Node B's clock: 12:00:00.450

A: sends message at A's time 12:00:00.500
B: receives at B's time 12:00:00.450      ← appears to receive before A sent it

NTP keeps drift to ~10ms but can't eliminate it. Clocks even go backwards during sync adjustments.

Lamport Clocks

Each process maintains a counter. Bump on event:

on local event:    counter += 1
on send msg:       counter += 1; attach counter to msg
on receive msg:    counter = max(counter, msg.counter) + 1

Properties:

• If event a happened-before b, then clock(a) < clock(b).
• Reverse not necessarily true — clock(a) < clock(b) doesn't mean a → b.
• Total order: ties broken by node ID.

A: counter 0 → 1 (local)
A: counter → 2 (send to B with timestamp 2)
B: receives, sets counter = max(0, 2) + 1 = 3

Useful for total-ordering events into a single log. Used in Cassandra (last-write-wins timestamps), older distributed locks.

Vector Clocks

Each process holds a vector of counters — one per process:

A: [Va, 0, 0]
B: [0, Vb, 0]
C: [0, 0, Vc]

on local event:    self[i] += 1
on send msg:       self[i] += 1; attach copy
on receive:        for each j: self[j] = max(self[j], msg[j]); self[i] += 1

Properties:

• a → b iff clock(a) ≤ clock(b) element-wise.
• a || b (concurrent) iff neither is ≤ the other.

Detects concurrent edits — critical for CRDTs, Dynamo-style conflict resolution.

Cost: O(N) size where N = number of nodes. Grows with cluster size.

Hybrid Logical Clocks (HLC)

Combines wall-clock and logical counters:

HLC = (physical_time, logical_counter)

on local event:
  if physical_time > hlc.physical:
    hlc = (physical_time, 0)
  else:
    hlc.logical += 1
  
on send: attach hlc
on receive (msg):
  hlc.physical = max(local_physical, msg.physical, hlc.physical)
  hlc.logical  = max(... computed appropriately) + 1

Properties:

• Close to wall-clock time (humans recognize timestamps).
• Captures happened-before via the logical part.
• Compact (16 bytes typical).

Used in CockroachDB, YugabyteDB, MongoDB causal consistency.

Comparison

Last-Write-Wins (LWW)

Cassandra and others: every write tagged with timestamp; on conflict, larger timestamp wins.

Problem: relies on wall-clock. Clock skew → "wrong" write wins. Lost data.

Mitigations:

• NTP discipline (still not perfect).
• HLC instead.
• Application-level vector clocks for conflict detection (then user resolution).

TrueTime (Google Spanner)

Spanner uses GPS + atomic clocks across DCs to bound clock uncertainty. TT.now() returns [earliest, latest] interval.

Commit waits out the interval before exposing the write — guarantees external consistency without coordination.

Hardware + careful engineering. Most don't replicate it.

Causal Consistency

A consistency model implementable with vector clocks or HLC:

op A → op B (causal)
all replicas see A before B
concurrent ops may be seen in either order

Strong enough for: collaborative editing, social feed ordering, replicated KV stores.

Idempotency vs Time

For idempotency by event ID, you don't need synchronized clocks — just unique IDs (Snowflake, UUID).

For ordering of events from the same producer, clocks suffice (just local). For cross-producer ordering, you need logical clocks.

Common Mistakes

[!NOTE] Wall clocks are advice; logical clocks are truth in distributed systems. If your design relies on "two machines agree on the current time," reconsider.

Interview Follow-ups

• "What's the difference between linearizability and causal consistency?" — Linearizable enforces real-time order on a single object globally. Causal enforces happened-before only; concurrent ops have flexible order.
• "How do you detect concurrent writes in Dynamo?" — Each write carries a vector clock; reads return all versions whose vectors are incomparable; application or LWW resolves.
• "How does Spanner avoid running 2PC for every read?" — TrueTime-bound timestamps + serializable snapshot isolation; reads from any replica at a given timestamp are linearizable.

Q: Unique ID generation — UUID, Snowflake, ULID, KSUID. Pick one.

Answer:

Every distributed system needs IDs. The naive auto_increment doesn't work across nodes; UUID v4 is random but unsorted. Modern choices like Snowflake, ULID, UUIDv7 combine uniqueness, sortability, and decentralization.

Requirements for a Good ID

• Unique: no collisions across distributed generators.
• K-sortable: rough time-ordering helps DB indexes and pagination.
• Compact: 16 bytes is fine; 64-bit is great.
• Stateless generation: no coordination per ID.
• Opaque externally: don't leak business semantics.

Options

UUID v4

hex: f47ac10b-58cc-4372-a567-0e02b2c3d479

128 bits, 122 of randomness. Collision probability negligible.

Cons:

• Not time-ordered → bad for DB index locality (random inserts → page splits in B-tree).
• Larger than 64-bit IDs.
• Hard for humans to glance.

UUID v7

018f8d34-... (first 48 bits = ms timestamp; rest random)

Released as draft-2024, in mainstream libs (Postgres uuid_generate_v7() via extensions).

Pros:

• 128 bits like UUID.
• Time-ordered: index locality, easy "newest first" sort.
• Drop-in replacement for UUID v4.

The modern default for new systems. Combines UUID's universality with sortability.

Snowflake

64-bit ID:

| 1 sign | 41 timestamp (ms) | 10 machine_id | 12 sequence |

• Timestamp: ms since epoch (Twitter's: 1288834974657).
• Machine ID: configured per node (10 bits = 1024 machines).
• Sequence: counter per ms per node.

Properties:

• 64-bit, compact.
• K-sortable (by timestamp).
• 4096 IDs/ms/machine = 4M IDs/sec/machine.
• Requires unique machine IDs.

Used by Twitter, Discord, Instagram (similar scheme).

Bookkeeping:

• Machine IDs from etcd / ZK / DB.
• Clock skew handling: refuse to generate if clock goes backwards.

ULID

128-bit, lexicographically sortable:

01H6FXM34VWCEK0NHJRA94QSEW
^^^^^^^^^^                  Crockford-base32 timestamp (48 bits)
          ^^^^^^^^^^^^^^^^   Random (80 bits)

Pros:

• Time-ordered.
• URL-safe.
• No coordination.
• Same size as UUID.

Library: lots; e.g., ulid in many languages.

KSUID (Segment)

20 bytes (160 bits):

27-char base62: 0o5sQohJzs8WW3hjwOyaCfaTeZB

Like ULID but bigger (more random bits → effectively no collision risk at any scale).

Sortability and DB Performance

B-tree primary key with random IDs (UUID v4):

• Each insert → random page in the index.
• Pages split, fragment, bloat.
• Cache locality poor.
• Insert-heavy workloads slow.

Sortable IDs (UUIDv7, ULID, Snowflake, auto_increment):

• Each insert → end of index (last page).
• Sequential writes; cache-friendly.
• 2–10× faster insert throughput in benchmarks.

For new systems: use sortable IDs.

Cross-Region / Multi-DC

• Snowflake: assign machine IDs per region (e.g., first 3 bits = region).
• UUIDv7 / ULID: no coordination at all.
• DB-managed auto_increment: doesn't work across regions.

Don't Expose Internal IDs

Sequential / sortable IDs leak:

• Total count (/orders/12345 → estimate Twitter's order volume).
• Creation order (privacy / business).

For external APIs: use opaque IDs (UUID or signed hashids). Internal: keep the sortable ones.

Choosing in 30 Seconds

External API URL identifier   → UUIDv4 or signed hashids
Internal primary key, new   → UUIDv7 or ULID
Already on Snowflake      → keep
Distributed numbering at extreme scale (10M+ IDs/sec)  → Snowflake-class
URL-friendly string         → ULID or NanoID

Common Mistakes

[!NOTE] For 2025 greenfield work, default to UUIDv7 unless you specifically need 64-bit (Snowflake) or URL ergonomics (ULID). UUIDv7 is what UUID v4 should have been.

Interview Follow-ups

• "How does Twitter generate IDs?" — Snowflake; each datacenter has a service that emits IDs with its assigned machine_id range.
• "What if a node's clock jumps backward?" — Refuse to generate (Twitter's choice); panic vs reject; tolerate small skew via NTP.
• "How do you migrate from int64 to UUID without downtime?" — Add UUID column nullable; backfill; flip writes to populate both; switch reads; drop old.

Q: Gossip protocols — what they are, where they're used.

Answer:

Gossip (epidemic) protocols spread state through a cluster by periodic peer-to-peer exchanges. They're decentralized, fault-tolerant, and scale to thousands of nodes — but eventually consistent.

How It Works

every second, each node:
  pick K random peers
  exchange state digests
  update local view with newer entries

Information spreads exponentially: O(log N) rounds to reach all nodes.

round 0:  1 node knows
round 1:  K nodes know
round 2:  K² nodes
...
round log_K(N): all nodes know

For K=3, N=1000: ~7 rounds.

Properties

• Decentralized: no leader.
• Fault-tolerant: lose any node; gossip continues.
• Scalable: each node talks to constant fanout regardless of cluster size.
• Eventually consistent: not linearizable.
• Probabilistic delivery: with overwhelming likelihood.

Use Cases

Anti-Entropy

The reconciliation step when two nodes exchange data and find disagreement:

Node A: { keyX: v3 }
Node B: { keyX: v2 }

A → B: "I have keyX = v3"
B: updates to v3

For large state, exchanging hashes (Merkle trees) reduces traffic. Cassandra uses this for SSTable repair.

SWIM Protocol

Specialized membership gossip (used by Consul, HashiCorp Serf):

1. Failure detection: probe random peer; if no reply, ask other peers to probe.
2. Dissemination: piggyback membership events on probes.

SWIM gives bounded failure-detection time + scalable.

Push vs Pull vs Push-Pull

• Push: I send my state to a peer. Effective when info is new.
• Pull: I ask the peer for theirs. Effective when info is established.
• Push-pull: both directions in one exchange. Most common.

Convergence Time

With K=3 peers per round and 1-second interval:

• 100 nodes: ~5 seconds to fully converge.
• 1000 nodes: ~7 seconds.
• 10000 nodes: ~9 seconds.

The "logarithmic-in-N" property is what makes gossip suitable for large clusters.

Failure Modes

Anti-Entropy vs Rumor-Mongering

Two modes:

• Anti-entropy: continuously reconcile entire state. Bandwidth-heavy; always converges.
• Rumor-mongering: spread only new info; stop after seeing it sufficient times. Bandwidth-light; small chance of missing.

Production systems mix: rumor for events, periodic anti-entropy for safety.

Limitations

• Eventually consistent: never use for "must be ordered" decisions.
• Bandwidth: chatty on big clusters; tune intervals.
• No durability: lost on restart unless persisted separately.
• Byzantine tolerance: only via stronger protocols (PBFT etc.).

Common Mistakes

[!NOTE] Gossip is the right tool for soft state: "who's in the cluster," "what's the cluster's view of itself," "node health." It's the wrong tool for any decision that must be exact.

Interview Follow-ups

• "Why does Cassandra use gossip?" — Membership + topology shared across all nodes; node count can be hundreds; no central coordinator desired.
• "Difference between gossip and pub-sub?" — Pub-sub: broker-centric, push only, often ordered. Gossip: peer-to-peer, eventual, no broker.
• "How fast can gossip detect a node failure?" — Configurable via SWIM probe intervals; typically 5–30 seconds depending on tuning.

Q: Redundancy, failover, multi-AZ/region — designing for failure.

Answer:

Every component will eventually fail. Reliability comes from redundancy (more than one of each thing) + failover (automatically reroute when one fails). Cost rises non-linearly: single-AZ → multi-AZ → multi-region → globally active-active.

Failure Domains

Anything that fails together is one failure domain:

• A single server.
• A rack.
• An availability zone (AZ).
• A region.
• A cloud provider.

Design so that one failure domain dying doesn't take down your service.

Single AZ (Not Enough)

• One AZ → AWS or GCP outage in that AZ takes you down (happens yearly).
• Not redundant against ZA/networking issues.

Multi-AZ

Spread across 2–3 AZs in the same region:

• Latency between AZs: ~1–2 ms.
• Cross-AZ bandwidth typically free or cheap.
• Survives single-AZ outage.

Standard for production:

LB (multi-AZ)
   │
   ├── app servers across AZ1, AZ2, AZ3
   ▼
DB primary in AZ1, replicas in AZ2, AZ3

Multi-Region

For regional disasters or compliance (data residency):

• Latency cross-region: 50–150 ms.
• Egress between regions: expensive.
• Cross-region sync replication: latency penalty.

Patterns:

1. Active-Passive (DR). Primary region serves all traffic. Standby is hot or warm. Failover triggers (manually or automatically) on regional outage.

RPO (data loss window) = replication lag. RTO (recovery time) = how long to switch.

2. Active-Active. Both regions serve traffic. Writes go to nearest. Replication is async.

Requires conflict resolution (multi-leader semantics). Best for partitioned data: each region owns its own user base.

3. Hot Standby with Read Replicas. Reads in any region; writes only in primary. Higher write latency for distant users.

Failover Mechanics

1. Detect: health checks fail consecutively (alert + auto-detect).
2. Verify: not a transient blip; confirm region is unreachable.
3. Promote standby DB to primary.
4. Update DNS / routing to point at standby.
5. Bring up app tier (or it was already warm).
6. Drain residual connections to old primary (if reachable).

Trade-off: aggressive failover causes flapping; conservative loses time.

DNS-Based vs Anycast Routing

DNS-based:

• TTL determines failover speed (30–60s minimum).
• Cached by resolvers, ignores TTL sometimes.

Anycast:

• Same IP advertised from multiple PoPs via BGP.
• Network routes to topologically nearest.
• Failover is sub-second.
• Used by Cloudflare, AWS Global Accelerator.

Data Replication Choices

Sync cross-region is brutal for write latency (100+ ms each). Most use async + accept the small RPO.

Cell-Based Architecture

Split users into cells, each a complete independent stack:

Cell 1: users [0–10M]    → own DB, app, cache
Cell 2: users [10M–20M]  → own DB, app, cache
...

Outage in Cell 5 doesn't affect Cell 3. Blast radius bounded.

Used by AWS, Stripe, Slack, Salesforce.

Chaos Engineering

You can't claim reliability without testing it. Chaos engineering:

• Inject failures in production (Netflix Chaos Monkey).
• Game-day simulations: take down an AZ during business hours.
• Validate runbooks: practiced or theoretical?

If you've never tested failover, it doesn't work.

Graceful Degradation

When dependencies fail, degrade rather than crash:

• Cache unavailable → fall back to DB (slower, still works).
• ML model service down → fall back to popular items.
• Email send fails → queue for retry.

See Graceful Degradation.

Per-Layer Redundancy

Failover Tests (Game Days)

Periodically practice:

• Kill the primary DB; measure failover time.
• Take down an AZ at the network level.
• Force a region failover; measure data loss.
• Verify runbooks work end-to-end.

Common Mistakes

[!NOTE] Reliability is a function of testing, not architecture diagrams. Until you've taken down each tier in production and seen the system recover, you don't know whether it works.

Interview Follow-ups

• "What's your RPO and RTO target?" — Per service, per tier. Recovery time vs data loss is a budget you spend.
• "How do you avoid cross-region sync writes for an active-active app?" — Partition by user/region; each region writes its own partition; replicate async.
• "How do you do cross-region capacity planning?" — Each region sized to handle its share + N/(N-1) when peer regions are out, to absorb failover traffic.

Q: Retries, backoff, jitter — done right.

Answer:

Retrying failed requests is essential. Done naively, retries amplify outages — the "retry storm." The recipe: exponential backoff + jitter + cap + circuit breaker.

What to Retry

Retry idempotent operations only. Non-idempotent: include Idempotency-Key.

Naive Retry (Don't)

while not success:
    response = call()
    if 5xx: continue       # immediate retry

If 100 clients all retry immediately, the server gets 100× load instantly. The thing that caused the failure (overload) gets worse.

Exponential Backoff

attempt 1: wait 100 ms
attempt 2: wait 200 ms
attempt 3: wait 400 ms
attempt 4: wait 800 ms
...
attempt N: wait min(2^N × base, max_backoff)

Spreads retries over time. Reduces aggregate load.

Add Jitter

Without jitter, all clients backoff identically and retry at the same instant:

delay = 2^attempt × base

Becomes:

delay = random(0, 2^attempt × base)         # full jitter

Or AWS's recommendation:

delay = min(max_backoff, random(base, 2^attempt × base))    # decorrelated jitter

Spreads retries across time; eliminates synchronized retry storms.

The Full Recipe

def call_with_retry(func, max_attempts=5, base=0.1, max_backoff=10):
    for attempt in range(max_attempts):
        try:
            return func()
        except RetryableException as e:
            if attempt == max_attempts - 1:
                raise
            delay = min(max_backoff, base * 2**attempt * random())
            time.sleep(delay)

Respect Retry-After

HTTP/1.1 429 Too Many Requests
Retry-After: 30

Server tells you when to come back. Honor it.

if response.status == 429:
    wait_sec = int(response.headers.get("Retry-After", 1))
    sleep(wait_sec)

Cap the Retries

Infinite retries → resource leak (connections, threads) when downstream is broken.

max_attempts = 5    # typical for synchronous user-facing requests
max_attempts = 20+  # for background jobs

After cap, propagate failure upward. Don't pretend it succeeded.

Retry at Which Layer?

Multiple retry layers stack and amplify:

client retry × LB retry × app retry × HTTP client retry = 16+ attempts per user click

Bad. Pick one layer to retry. Other layers should fail fast.

Typical: HTTP client library retries (with reasonable defaults); other layers don't.

Circuit Breaker

After many failures in a row, stop calling for a while:

state: CLOSED (calls allowed) → OPEN (calls fail fast) → HALF_OPEN (probe a few)

See Circuit Breaker.

Pair with retry: retry helps transient failures; circuit breaker handles sustained failure.

Hedged Requests

For latency optimization, not failure handling:

t=0: send request to A
t=p95: still no response → send to B
t=any: take first response, cancel other

Used by Google for tail-latency reduction. Doubles load on slow operations.

Retry Storm

Classic outage pattern:

Service A is slow.
Clients of A retry.
A gets 3× normal load.
A becomes slower / starts failing.
Clients retry more.
A is now totally overwhelmed.

Mitigations:

• Aggressive timeouts (fail fast).
• Limited retries.
• Circuit breaker.
• Backoff + jitter.
• Load shedding by A itself.

Retries vs Backoff in Real Systems

HTTP client (e.g., reqwest, axios):
  retries: 3
  backoff: exp with jitter
  max delay: 5s

Async queue (Kafka consumer):
  retries: infinite (or N to DLQ)
  backoff: exp with cap 5 min

DB driver:
  retries: 2-3
  on serialization-failure: retry with backoff

Idempotency Is Mandatory

Retries that aren't idempotent corrupt state.

For HTTP POST: Idempotency-Key header → server dedups. For DB writes: unique constraint on natural key. For messages: dedup by event_id.

See Idempotency.

Common Mistakes

[!NOTE] The most operationally important retry pattern is "exponential backoff with full jitter." Memorize it; apply it everywhere. The default in libraries is usually NOT this — set it yourself.

Interview Follow-ups

• "What's the difference between full and decorrelated jitter?" — Full jitter samples uniformly over the backoff window. Decorrelated jitter widens with each attempt; better avoids synchronized retries.
• "How do you handle retries with stream processors?" — Tiered retry topics (Kafka) — failed events flow into delay topics; eventually DLQ.
• "What's a 'retry budget'?" — Cap on the rate of retries the system as a whole tolerates. If retries exceed N% of total traffic, stop retrying. Used by Envoy.

Q: Circuit breaker, bulkhead, timeout — failure-isolation patterns.

Answer:

Three patterns from the fault-tolerance toolbox, each protecting different things:

• Timeout: bound how long you wait.
• Circuit breaker: stop calling a failing dependency.
• Bulkhead: isolate resources so one failure doesn't sink everything.

Timeout

The simplest, most under-used pattern.

client.get(url, timeout=2s)
db.query(sql, timeout=500ms)

Without timeouts:

• Slow downstream stalls your threads.
• Threads exhaust → can't serve any request.
• Cascading failure.

Pick timeouts by:

• Median + 3× standard deviation of normal.
• p99 of healthy + margin.
• Aggressive enough that a stuck downstream doesn't wait forever.

Default: 1–3 seconds for HTTP; 100–500 ms for in-DC RPC.

Circuit Breaker

State machine:

CLOSED (normal):
  pass calls through; track success/failure ratio
  if failure rate > threshold → OPEN

OPEN:
  fail fast — return error without calling
  after wait period → HALF_OPEN

HALF_OPEN:
  pass a few probe calls
  all succeed → CLOSED
  any fail → OPEN

Why: when downstream is broken, retrying is wasteful AND amplifies the outage. Stop calling.

Library implementations: Hystrix (deprecated), resilience4j (JVM), polly (.NET), sony/gobreaker (Go), opossum (Node).

Configuration:

• failure_threshold: 50% over last N calls.
• sliding_window: e.g., 100 calls or 10 seconds.
• open_duration: 30 seconds typical.
• half_open_probes: 5 calls.

Bulkhead

Borrowed from ship design: walls between sections so flooding one doesn't sink the ship.

In services: isolate resources per dependency.

threadpool[orders]:    100 threads
threadpool[catalog]:    50 threads
threadpool[email]:      20 threads

Catalog goes slow → only catalog threads block; orders thread pool unaffected.

Without bulkheads, all calls share one pool. Catalog slowness starves orders.

Variants:

• Thread pool per dependency (Hystrix style).
• Semaphore per dependency (lighter; no thread overhead).
• Connection pool per dependency (DB, HTTP).

Combined Pattern

client → timeout → circuit breaker → bulkhead → call dependency

1. Bulkhead caps concurrency.
2. Circuit breaker fails fast when dependency unhealthy.
3. Timeout bounds individual call duration.

All three together = isolation depth.

Implementation Example (Resilience4j)

@CircuitBreaker(name = "paymentsApi", fallbackMethod = "fallback")
@TimeLimiter(name = "paymentsApi")
@Bulkhead(name = "paymentsApi")
@Retry(name = "paymentsApi")
public CompletableFuture<PaymentResp> charge(ChargeReq req) {
    return client.post(...);
}

PaymentResp fallback(ChargeReq req, Throwable t) {
    return PaymentResp.queued(req);
}

Order matters: outer to inner is Retry → Breaker → RateLimiter → Bulkhead → TimeLimiter.

Observability

• CB state: per dependency, time spent in each state.
• Bulkhead saturation: in-use vs max permits.
• Timeout rate: how often timeouts fire.

Alert on:

• CB in OPEN state for > N minutes.
• Bulkhead near saturation.
• Timeout rate > threshold.

Choosing Thresholds

Too tight: false-positive opens during normal load spikes. Too loose: opens too late, damage done.

Start at 50% failure rate over a meaningful window (50–100 calls). Tune from data.

Common Mistakes

When Not To Use Each

Timeout: always use. There's no excuse.

Circuit breaker: skip for non-critical path that you'd retry anyway. Don't break on rare network blips.

Bulkhead: overkill for monoliths with a single dependency. Required for services calling many.

Pattern Composition for Production Service

Per dependency:
  Bulkhead: 50 concurrent permits
  Timeout: 2 seconds
  Circuit breaker: 50% failure over 100 calls / 30s window
  Retry: 3 attempts with jitter (only on idempotent)
  Fallback: cached response or sensible default

Track each layer's metrics.

[!NOTE] These patterns aren't magic — they protect against the specific failure of "downstream slow / failing." For other failures (bug in your service, data corruption), they don't help. Mix with health checks, runbooks, capacity planning.

Interview Follow-ups

• "What's the difference between a circuit breaker and a rate limiter?" — Breaker reacts to failure rate. Limiter caps QPS regardless of success. Different protections, often used together.
• "How does a circuit breaker know when to close?" — HALF_OPEN state allows a few probes; success → CLOSED; failure → back to OPEN.
• "What's load shedding?" — Service deliberately rejects requests when overloaded (often based on queue size or RT). See Graceful Degradation.

Q: Idempotency — what it is, why it's mandatory, how to enforce it.

Answer:

An operation is idempotent if performing it multiple times has the same effect as once. In a world of retries, networks that lie, and at-least-once delivery, idempotency isn't optional — it's the only way to guarantee correctness.

Why It Matters

Networks fail. Clients retry. Brokers redeliver. Without idempotency:

• Double-charges.
• Duplicate orders.
• Counters off by 5×.
• Email sent twice.
• Records inserted N times.

Idempotency turns at-least-once delivery into exactly-once effect.

Naturally Idempotent Operations

• SET balance = 100: same value → same state.
• DELETE WHERE id = 5: gone is gone.
• PUT /resource/123 (replacement).

Naturally NON-Idempotent

• INCREMENT count: each call adds.
• INSERT new row: each call adds a row.
• SEND email: each call sends.
• POST /charges (creates): each call creates.

These need explicit dedup.

Patterns to Make Operations Idempotent

1. Idempotency Keys

Client sends a unique key per logical operation:

POST /charges
Idempotency-Key: 8a4b8c3e-...
{ "amount": 100 }

Server stores result keyed by this. Replay returns the cached result.

def charge(req, idempotency_key):
    cached = store.get(idempotency_key)
    if cached:
        return cached.response

    result = do_actual_charge(req)
    store.put(idempotency_key, result, ttl="24h")
    return result

The key is client-generated, scoped to (account, key). Same key → same result.

See Idempotent REST APIs.

2. Unique Constraints in DB

CREATE TABLE orders (
    id UUID PRIMARY KEY,
    client_order_id UUID UNIQUE,    -- client-provided
    ...
);

INSERT INTO orders ... ON CONFLICT (client_order_id) DO NOTHING;

Duplicate insert → no-op via constraint.

3. Conditional Updates

UPDATE accounts SET balance = balance - 100
WHERE id = ? AND balance >= 100 AND last_tx_id <> ?

Use a last_tx_id or version field; idempotent because second call doesn't match.

Or simpler with optimistic concurrency:

UPDATE row SET ..., version = version + 1
WHERE id = ? AND version = ?

4. Event Sourcing / Inbox

Consumer dedups by event ID:

INSERT INTO inbox (event_id) VALUES (?) ON CONFLICT DO NOTHING;
-- if conflict: ignore
-- else: apply

Pairs with at-least-once delivery to give exactly-once effect.

Idempotency Storage

Where to keep the key → result mapping:

For payments: same DB as the financial data is the only correct answer.

TTL on Idempotency Records

Records grow forever otherwise. Stripe's choice: 24 hours.

After TTL, a new replay with the same key is treated as a new request. Acceptable because retries usually happen within minutes.

For longer retention (audit), keep a compact log separately.

HTTP Methods and Idempotency

Per spec:

• GET, PUT, DELETE: idempotent.
• POST, PATCH: not.

In practice, POST is the action that creates side effects — and the one that absolutely needs Idempotency-Key.

Async Pipelines

Same principle in event-driven:

producer → at-least-once delivery → consumer
consumer dedups by event_id

Without this, every retry = duplicate effect.

Exactly-Once vs Idempotent

"Exactly-once delivery" = impossible over an unreliable network (Two Generals Problem).

"Exactly-once effect" = achievable: at-least-once delivery + idempotent consumer.

The difference is critical in interviews. Don't say "exactly-once delivery" with a straight face.

Common Mistakes

Validation Pattern

Idempotency record + atomic business write:

BEGIN;
INSERT INTO idempotency_records (key, status='in_progress', client_id=?)
       VALUES (?, 'in_progress', ?)
ON CONFLICT (key) DO NOTHING;
-- if conflict: return existing record's response

INSERT INTO orders (...);

UPDATE idempotency_records SET status='completed', response=? WHERE key=?;
COMMIT;

Returns same response on retry; never creates two orders.

[!NOTE] Every external write endpoint should accept Idempotency-Key. Every async consumer should dedup by event ID. Build it into your framework defaults; otherwise it gets forgotten and bites later.

Interview Follow-ups

• "What if two distinct operations happen to use the same key?" — Scope keys per (tenant, client, request). Or accept "first wins" semantics; document it.
• "How do you handle slow first-call + concurrent retry?" — Insert idempotency record at "in_progress" state with TTL; second call sees in_progress → return 409 or wait.
• "What's the difference between idempotency and 2PC?" — 2PC tries to make multiple steps atomic. Idempotency tries to make one step safe-to-replay. The latter is much cheaper.

Q: Graceful degradation and load shedding — when failure beats unavailability.

Answer:

Under stress, a system can do one of three things:

1. Maintain full service (best — only if capacity allows).
2. Degrade gracefully (serve a reduced version of the feature).
3. Crash / hang (worst — affects everyone).

The pattern: serve a worse experience to many rather than fail a few completely.

Graceful Degradation Examples

User sees something. They don't see an error.

Load Shedding

Reject excess requests at the entry point to protect the system from collapse.

healthy:    accept all
degraded:   accept critical, reject low-priority
overloaded: accept only health checks; return 503

Mechanism:

• Queue depth check at gateway: if > N, return 503.
• Per-tenant quotas: drop excess from one bad actor.
• Adaptive load shedding: shed proportional to observed latency.

Drop early, drop fast. Letting requests queue makes everyone slow.

Tiered Response

Categorize requests:

• Critical: user-paying actions, security, authentication. Never shed.
• Important: core feature use. Shed last.
• Optional: analytics, marketing, recommendations. Shed first.

Tag at the gateway / load balancer.

Adaptive Concurrency

Borrowed from TCP: AIMD (Additive Increase, Multiplicative Decrease).

in-flight requests = N
observe response time
if RT good: N += 1
if RT bad:  N = N × 0.7

Auto-tunes capacity per downstream. Used by Netflix concurrency-limits, Envoy adaptive concurrency.

Failure Modes for Dependencies

When dependency X fails:

Circuit breaker + fallback is the typical implementation.

Cache-as-Fallback

def get_recommendations(user):
    try:
        return ml_service.recommend(user)
    except ServiceUnavailable:
        return popular_items_cache.get()

When ML service is broken, return popular items instead. UX slightly worse; service still works.

"Read-Only Mode"

For DB writes: when primary unhealthy, serve reads from replica; reject writes with clear UX.

GET /users/me            → works (replica)
POST /orders             → 503 "we're temporarily unable to take new orders, please retry"

Better than crashing.

Feature Flags as Kill Switches

Per-feature toggles let you disable a broken feature without redeploying:

if flag("recommendations_enabled"):
    show_recs(user)

Flip the flag → recommendations disabled across all instances within seconds.

Used in incidents to isolate damage.

Backpressure as Degradation

When a downstream is slow, applying backpressure to clients (return 429 with Retry-After) is degradation:

client should retry later
service stays alive

See Backpressure.

Communicating Degradation

Don't pretend everything's fine:

401 → "you're not logged in"
503 → "high traffic, retry in 30 seconds"
"Some features temporarily unavailable" banner

Users tolerate degradation if you tell them. They don't tolerate mystery slowness.

Detection / Triggers

• Latency > threshold.
• Error rate > threshold.
• Queue depth > N.
• Downstream circuit breaker open.
• Manual override (oncall flips a flag).

Automated triggers good for fast response; manual triggers for cases that need judgment.

Recovery

When pressure subsides, re-enable degraded features:

• Slow ramp (canary back to full).
• Test each tier as it comes back.
• Don't restore instantly — that often re-triggers the original issue.

Common Mistakes

[!NOTE] Design degraded modes during peace time, not during incidents. The question to ask of every dependency: "if this is down, what does the user see?" If the answer is "an error," design something better.

Interview Follow-ups

• "How do you decide between failing and degrading?" — If the action is critical (payment), fail clearly. If it's auxiliary (recs), degrade silently.
• "How do you test graceful degradation?" — Game days; chaos engineering injects dependency failures and verifies UX.
• "How do you avoid 'always degraded' state?" — Alert on degradation duration; force a postmortem if a fallback fires for more than N minutes.

Q: CQRS — Command-Query Responsibility Segregation.

Answer:

CQRS splits the write model (commands) from the read model (queries). Each is optimized for its access pattern, often using different schemas, stores, and even consistency levels. It pairs naturally with event-driven systems and is the easiest path to scaling read-heavy workloads.

Without CQRS

One model, one schema, one DB:

        Service
   │             │
   write API     read API
       │           │
       ▼           ▼
   ┌─────────────────┐
   │   one table     │   <- shape compromises both
   └─────────────────┘

Writes need normalized rows; reads need denormalized, pre-joined data. The single schema is a compromise that satisfies neither.

With CQRS

Two models:

   Commands              Queries
       │                    │
       ▼                    ▼
   Write side          Read side
   (source of truth)   (denormalized views)
       │                    ▲
       │  events / CDC      │
       └────────────────────┘

• Write side: small, normalized, transactional.
• Read side: denormalized, pre-joined, query-shaped.
• Synced asynchronously via events or CDC.

Concrete Example

E-commerce order tracking.

Write side (Postgres):

orders   (id, user_id, status, ...)
items    (order_id, sku, qty, price)

Strict ACID, used by POST /orders, PATCH /orders/:id.

Read side (Elasticsearch + Redis):

// One denormalized doc per order, indexed by user
{
  "order_id": "o123",
  "user_id":  "u42",
  "status":   "shipped",
  "items":    [{ "sku": "abc", "name": "Widget", "qty": 2, "subtotal": 19.98 }],
  "total":    19.98,
  "shipped_at": "2025-...",
  "tracking_url": "..."
}

Used by GET /users/:id/orders and GET /orders/:id. One query returns everything the UI needs.

Sync: on every write, publish events; a projection updates Elasticsearch.

Where Each Side Lives

You can also stay in one DB and use materialized views as the read model — lightweight CQRS without polyglot.

When CQRS Is Right

• Read patterns very different from write (e.g., complex aggregations).
• Reads >> writes (10:1, 100:1).
• Need multiple read shapes (dashboard, mobile, admin) without each pounding the write DB.
• Already event-driven.

When CQRS Is Wrong

• CRUD apps with simple, symmetric read/write.
• Strong read-your-writes required everywhere (CQRS's read side lags).
• Team can't afford the operational complexity.

CQRS adds eventual consistency to a path that was previously instant. Make sure the UX can absorb it.

Read-Your-Writes Challenges

After POST /orders, the user immediately refreshes. The read store hasn't been updated yet → user sees no order.

Mitigations:

1. Optimistic UI: client renders the just-written value locally.
2. Read-through cache: write side updates a fast cache synchronously, slow read store async.
3. Version pinning: client sends min_version=N; server waits or routes to write DB until projection catches up.
4. Hybrid: critical reads from write DB, list/search reads from projection.

Eventual Consistency Story

Write happens; projection updates after a delay (50ms – several seconds typically). Measure:

projection_lag = now() - latest_processed_event_timestamp

Alert at thresholds; show "syncing" UI; gate auditable actions on linearizable reads.

CQRS Without Event Sourcing

CQRS is often confused with Event Sourcing. They're separate:

• CQRS: separate read/write models.
• Event Sourcing: log of events is the source of truth (write side stores events, not state).

You can do CQRS with a traditional write side (just publish events for the projection). See Event Sourcing.

Projection Patterns

A projection is the code that builds the read model from events.

on OrderPlaced(id, user_id, items):
    upsert read_orders (id, user_id, items, status='placed')

on OrderShipped(id, tracking):
    update read_orders set status='shipped', tracking_url=... where id=...

on OrderCancelled(id):
    update read_orders set status='cancelled' where id=...

Properties:

• Idempotent (replays don't break).
• Eventually consistent.
• Re-buildable: nuke the read store, replay all events from offset 0 → rebuild from scratch.

That last property is gold. Bug in projection logic? Fix code, rebuild from event log. Want a new read shape? Add a projection, replay history.

Side Reads on Write Side

A common variant: simple/critical reads (GET /orders/:id right after POST) hit the write side; expensive/list reads (GET /users/:id/orders?status=...) hit the projection.

                    ┌── write tier (Postgres)
GET /orders/:id ────┤  read-after-write reads
                    │  occasional simple reads
POST /orders ───────┘

GET /search ────────► read tier (Elastic)

Best of both: tight consistency where it matters; scale where the queries are.

Common Mistakes

[!NOTE] CQRS shines when read and write demands are genuinely different. If they're the same, you're paying complexity for nothing. Use it where it pays back.

Interview Follow-ups

• "How is CQRS different from a read replica?" — Replica has the same schema. CQRS allows different schema, different DB engine, different consistency.
• "How do you keep projections in sync with new code?" — Version them; deploy in parallel; flip traffic when caught up; old projection deleted after burn-in.
• "What's the projection failure recovery?" — Resume from last committed offset; idempotent operations → safe replay.

Q: Event Sourcing — when is "store every event" worth it?

Answer:

Event Sourcing stores the sequence of events that produced state, not the state itself. State is derived by replaying events. Powerful pattern with high ceremony — right for some domains (banking, audit), overkill for most.

The Core Idea

Traditional model:

accounts table:
   id | balance
   42 | 1000

Event-sourced model:

events log:
   id=1  type=AccountOpened    account=42  balance=0
   id=2  type=Deposited        account=42  amount=500
   id=3  type=Deposited        account=42  amount=600
   id=4  type=Withdrawn        account=42  amount=100

derived: balance = 0 + 500 + 600 - 100 = 1000

The events are immutable. The balance is a projection derived by replaying them.

Properties

• Full audit trail. Every change is a permanent record with who/when/why.
• Time travel. Replay to any point: "what was the balance on July 4?"
• Multiple read models. Same event log → many projections.
• Reproducible bugs. Bug in derivation? Replay from log with fixed code.
• Natural CDC source. Other systems can subscribe to the event stream.

The Pieces

Commands ──► Aggregate ──► Events ──► Event Store ──► Projections / Read Models
                              │                              │
                              ▼                              ▼
                       publish to bus              query views (CQRS)

• Command: intent ("transfer $100").
• Aggregate: domain object validating the command + emitting events.
• Event: fact ("transferred $100"). Past tense, immutable.
• Event Store: append-only log keyed by aggregate ID.
• Projection: process consuming events to build read views.

Aggregate Pattern

class Account:
    def __init__(self, events):
        self.balance = 0
        self.id = None
        for e in events:
            self.apply(e)

    def apply(self, event):
        if event.type == "AccountOpened":
            self.id = event.account
        elif event.type == "Deposited":
            self.balance += event.amount
        elif event.type == "Withdrawn":
            self.balance -= event.amount

    def withdraw(self, amount):
        if amount > self.balance:
            raise InsufficientFunds()
        return Event("Withdrawn", account=self.id, amount=amount)

Load: read events, apply. Command: emit event. Save: append event.

Snapshots

Replaying every event from year zero gets slow at high counts. Snapshot the aggregate state every N events:

load(account_id):
    snapshot = load_latest_snapshot(account_id)
    events = load_events_after(snapshot.version)
    return apply_all(snapshot, events)

Snapshots are an optimization, not the source of truth. Throw them away anytime; they regenerate.

Schema Evolution

Events are forever. You can't change one — only add new ones. Strategies:

• Add new field with default (Avro/Proto backward-compatible).
• Upcasters: code that rewrites old event versions to new shapes at load time.
• Never delete. Mark "deprecated" instead.

A subtle bug: a 2017 event of type AddressUpdated won't have a country field you added in 2023. Your replay code must handle that.

When Event Sourcing Is Right

• Compliance / audit-heavy domains (banking, healthcare, legal).
• "What changed and when" is a primary query.
• Domain has clear, stable business events (already DDD-modeled).
• You want eventual derivation of multiple read models.
• Replay-based recovery is valuable.

When It's Wrong

• Pure CRUD ("update users.email"); events would be EmailChanged and nothing else interesting.
• Team unfamiliar with the pattern.
• High write throughput on a single aggregate (every event is a serialization point).
• Reporting needs ad-hoc SQL across all entities at once (projections rescue this, but with effort).

Implementation Options

Eventual Consistency

Same gotcha as CQRS. Projections lag the event store. Read-your-writes patterns:

• Optimistic UI.
• Query the latest snapshot + recent events.
• Pinned read after write for short window.

Concurrency Control

Two commands on the same aggregate race. Use optimistic concurrency:

load events at version 42
build aggregate state
emit new event(s)
append IF current version == 42
   on mismatch: reload + retry

Optimistic version checks at append; conflicts trigger retry. No locks held.

Sagas Often Live Above Event Sourcing

Inter-aggregate workflows: events from one aggregate trigger commands on another. The orchestrator (or choreography) lives outside the aggregate boundary. See Saga.

Cost of Event Sourcing

• Storage: O(events) forever. 10× more than current-state storage; with compression, still ~3×.
• Cognitive load: every change starts with "what event is this?"
• Migration: schema versioning is non-trivial.
• Tooling: rebuild from offset 0 is your safety net; needs to actually work.

Patterns It Enables

• Time-travel debugging: replay to the exact moment a bug appeared.
• A/B projection: build two read models from same events for migration.
• Audit trail: free, complete, queryable.
• Temporal queries: "what did this look like at midnight last Friday?"
• External integration: external system gets a clean event stream, no DB CDC kludges.

Common Mistakes

[!NOTE] Event Sourcing is the strongest, slowest version of "audit-ability." If the domain doesn't need it strongly, a normal DB + outbox is usually a better trade.

Interview Follow-ups

• "Difference between Event Sourcing and CDC?" — ES emits business events from the domain layer (intent + meaning). CDC emits database-row changes (after the fact). ES has semantics; CDC has bytes.
• "How do you delete data (GDPR)?" — Crypto-shredding: encrypt PII with a per-user key, delete the key on request → data unrecoverable. Or null out specific fields in old events ("tombstone").
• "How do you replay 100M events?" — Snapshot + replay only after; parallelize per aggregate; back-pressure the projection rebuilder.

Q: Saga pattern — choreography vs orchestration. When and how?

Answer:

A Saga manages a multi-step business transaction across services without distributed 2PC. Each step is a local transaction; failures trigger compensating actions to undo earlier steps. Two flavors: choreography (event-driven, no coordinator) and orchestration (central coordinator drives the flow).

The Problem

Place order = {
  reserve inventory   (service A)
  charge card         (service B)
  ship                (service C)
  notify              (service D)
}

A relational transaction across A, B, C, D would need 2PC (XA) — slow, fragile, doesn't work across HTTP. Saga decomposes into 4 local transactions + 4 compensations.

Choreography (Event-Driven)

Each service publishes events when its step completes; other services subscribe and react.

order-service ──OrderPlaced──► inventory-service
                                  │
                                  ▼
inventory-service ──InventoryReserved──► payment-service
                                            │
                                            ▼
payment-service ──PaymentCharged──► shipping-service
                                       │
                                       ▼
                          ShipmentScheduled

On failure:

payment-service ──PaymentFailed──► inventory-service
                                       │
                                       ▼
                              compensate: release stock
                              publish: InventoryReleased
                                       │
                                       ▼
order-service ──OrderFailed──► notify-service

Pros:

• No central component.
• Services loosely coupled.
• Easy to add new participants.

Cons:

• Hard to reason about the full flow.
• Hard to debug (where is my order stuck?).
• Cyclic event dependencies easy to introduce.
• Adding compensations means changes scatter across services.

Orchestration (Central Coordinator)

A saga orchestrator explicitly calls each step and decides what to do on failure.

Orchestrator
   │ call inventory.reserve()
   │ ◄── reserved
   │ call payment.charge()
   │ ◄── failure
   │ call inventory.release()       ← compensation
   │ ◄── released
   │ publish OrderFailed

Implemented with: Temporal (or Cadence), AWS Step Functions, Camunda, Netflix Conductor, or hand-rolled state machines.

Pros:

• Workflow is explicit, version-able, debuggable.
• Centralized error handling.
• Easier to test end-to-end.
• Built-in retries, timeouts, signal handling.

Cons:

• Orchestrator is itself a service to operate.
• Couples services to the orchestrator's view of the world.
• Can become a "god class" if not carefully scoped.

When to Pick Which

Many production systems mix: orchestrator for primary flow, events for side effects.

Compensation Semantics

A compensation is not a rollback — it's a business-level undo.

Step                        Compensation
charge card                 refund (with original tx id)
reserve inventory           release reservation
send email                  send "actually, cancel that" email

The compensating action must be idempotent — it might run more than once (retries, replay). Track state explicitly (reservation_released = true) and no-op on repeat.

Some actions are not compensable: "send SMS to user" can't be unsent. Design for forward-only flow at those points, or move them after the point of no return.

Idempotency Is Mandatory

Saga steps are at-least-once. Every step must dedup by saga ID + step ID:

service.reserve(orderId="o123", step="reserve_v1")
   if step already processed → return cached result
   else                       → do work, record step

Stripe, Temporal, Step Functions: all enforce this with deterministic step IDs.

Saga State

Where the orchestrator (or aggregate of services in choreography) tracks progress.

saga_id        VARCHAR PK
current_step   ENUM
status         ENUM (in_progress, completed, failed, compensating)
context        JSONB        -- inputs, intermediate results
created_at     TIMESTAMPTZ
updated_at     TIMESTAMPTZ

Recovery: on orchestrator restart, load all in_progress sagas and resume.

Saga + Outbox

To publish events atomically with a DB write, use the Transactional Outbox pattern:

BEGIN;
INSERT INTO orders (...);
INSERT INTO outbox (event_type, payload) VALUES ('OrderPlaced', '{...}');
COMMIT;

Separate process polls outbox (or CDC via Debezium) → publishes to Kafka. Atomic with the order write. No "we committed but the event didn't fire" bug.

See Transactional Outbox.

Failure Modes

Visualizing

ORDER PLACED
   │
   ▼
[Reserve Inventory]──fail──►[Release Inventory]
   │ ok                       │
   ▼                          ▼
[Charge Card]──fail──►[Refund + Release Inventory]
   │ ok                       │
   ▼                          ▼
[Schedule Shipment]──fail──►[Refund + Release + cancel?]
   │ ok                       │
   ▼                          ▼
COMPLETED                  FAILED

Hand-drawing this diagram in an interview is half the answer.

Common Mistakes

[!NOTE] Sagas trade ACID for availability + service autonomy. The price: every business invariant becomes an explicit compensation. Design the compensations first; the happy path is the easy part.

Interview Follow-ups

• "Why not 2PC?" — Blocks resources on participants; coordinator failure leaves locked rows; doesn't work over WAN.
• "How do you implement Temporal-style workflows without Temporal?" — Persistent state machine in DB + idempotent step handlers + scheduler. Re-inventing Temporal, badly.
• "What about sagas with parallel branches?" — Both choreography and orchestration support fan-out; orchestrator must wait for all branches before next step (Promise.all semantics).

Q: Transactional Outbox — atomic DB write + event publish.

Answer:

You write to your DB and need to publish a message — to Kafka, SNS, another service. The naïve "write then publish" loses events when the process crashes between the two steps. The outbox pattern solves this with one DB transaction.

The Failure Mode

order = insert_order()        # ✅ committed
publish("OrderPlaced", order) # ❌ process crashed
                              #    DB has order, no event sent

# OR

publish("OrderPlaced", order) # ✅ sent
order = insert_order()        # ❌ failed
                              #    event refers to non-existent order

Either order, dual writes can diverge. No clever try/catch fixes it — the two systems aren't transactional together.

The Solution

Write the event to an outbox table in the same DB in the same transaction. A separate process drains the outbox and publishes:

BEGIN;
INSERT INTO orders   (id, ...)                                VALUES (?, ...);
INSERT INTO outbox   (id, aggregate_type, event_type, payload, created_at)
                     VALUES (?, 'Order', 'OrderPlaced', ?, now());
COMMIT;

Either both rows are present or neither is. Atomic.

A relay process polls outbox, publishes, marks as sent:

SELECT * FROM outbox WHERE published_at IS NULL ORDER BY id LIMIT 100;
publish to Kafka
UPDATE outbox SET published_at = now() WHERE id IN (...);

Outbox Schema

CREATE TABLE outbox (
    id              BIGSERIAL PRIMARY KEY,
    aggregate_type  TEXT NOT NULL,
    aggregate_id    TEXT NOT NULL,
    event_type      TEXT NOT NULL,
    payload         JSONB NOT NULL,
    headers         JSONB,
    created_at      TIMESTAMPTZ DEFAULT now(),
    published_at    TIMESTAMPTZ
);

CREATE INDEX ON outbox (published_at, id) WHERE published_at IS NULL;

The partial index makes polling cheap.

Two Relay Strategies

1. Polling

A worker reads unpublished rows, sends them, marks them sent.

while True:
    rows = db.fetch("SELECT * FROM outbox WHERE published_at IS NULL ORDER BY id LIMIT 100")
    if not rows:
        sleep(0.1)
        continue
    for r in rows:
        kafka.send(r.event_type, r.payload)
    db.exec("UPDATE outbox SET published_at = now() WHERE id IN ...")

Simple. Adds latency (polling interval). Concurrent workers can race — use FOR UPDATE SKIP LOCKED to safely parallelize:

SELECT * FROM outbox
WHERE published_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;

2. Change Data Capture (CDC)

Tail the database's replication log (WAL/binlog). Tools: Debezium, Maxwell, AWS DMS.

Postgres WAL
    │
    ▼
Debezium connector
    │
    ▼
Kafka topic "outbox.events"
    │
    ▼
Downstream consumers

Sub-second latency, no polling load on the DB. Operationally heavier (Debezium cluster, schema management).

CDC bonus: it can stream changes from any table, not just outbox. Some systems skip the outbox table and CDC the business tables directly — works if events can be derived from row changes alone.

Idempotent Publish

Even with outbox, at-least-once delivery is the rule. The relay might:

• Send to Kafka but crash before marking sent → resends on retry.
• Mark sent but Kafka acked late → next worker picks up.

Consumers must dedup by outbox.id or a business event ID. Standard idempotency pattern.

Ordering

Per aggregate_id ordering preserved if you partition outbox dispatch by it:

publish to topic "orders" with key = aggregate_id
                              → Kafka guarantees per-partition order

Across aggregates, no order guarantee. That's usually fine; design events to be commutative or carry timestamps.

Cleanup

Outbox grows forever if you don't prune:

DELETE FROM outbox WHERE published_at < now() - INTERVAL '7 days';

Keep enough history to replay if something downstream crashes. 7–30 days typical.

For very high volume: partition outbox by day (Postgres declarative partitioning) and drop old partitions.

Outbox vs Other Patterns

Outbox + Inbox

Mirror pattern on consumer side: store inbound events in an inbox table before processing, in the same transaction as the business write:

BEGIN;
INSERT INTO inbox (event_id, event_type) VALUES (?, ?);   -- unique constraint dedups
INSERT INTO orders (...);                                  -- business work
COMMIT;

If event is replayed: unique constraint violation on event_id → handler safely no-ops. Bullet-proof at-least-once → exactly-once-effect.

Real-World Example

E-commerce checkout:

BEGIN;
  INSERT INTO orders (id, user_id, total, status='pending');
  INSERT INTO outbox (event_type='OrderCreated', payload={...});
COMMIT;

Debezium → Kafka → consumers:
  - payment-service:   charge card
  - inventory-service: reserve stock
  - email-service:     send confirmation
  - analytics-service: record event

Each consumer dedups by event_id (from outbox).

Common Mistakes

[!NOTE] Outbox isn't optional for any system that writes to a DB and publishes events. The "dual write" alternative looks fine in dev and silently loses events in production.

Interview Follow-ups

• "Why not just use a transactional Kafka producer?" — KIP-98 Kafka transactions span Kafka topics, not your DB. You'd need XA-style 2PC; not natively supported by most DBs.
• "What about NoSQL?" — Same idea works if your NoSQL supports per-document/transaction atomicity. DynamoDB Streams (CDC) + outbox attribute works.
• "How would you ship events ordered globally?" — You usually don't need that. Per-aggregate ordering via partition key is enough. If you really need it, you've designed yourself into a single-writer bottleneck.

Q: Change Data Capture (CDC) — what it is, when to use it.

Answer:

CDC streams every row change in a database as an event. Built originally for data warehousing; now the backbone of microservice integration, materialized views, audit, and search indexes.

What It Looks Like

DB write:                            CDC event:
INSERT INTO orders (id, total)    →  {
  VALUES (123, 50)                    "op": "create",
                                       "table": "orders",
                                       "after": {"id": 123, "total": 50}
                                     }

UPDATE orders SET total=60         →  {
  WHERE id=123                        "op": "update",
                                       "before": {"id": 123, "total": 50},
                                       "after":  {"id": 123, "total": 60}
                                     }

DELETE FROM orders WHERE id=123    →  {
                                       "op": "delete",
                                       "before": {"id": 123, "total": 60}
                                     }

How It Works

DBs maintain a transaction log (Postgres WAL, MySQL binlog, MongoDB oplog) for replication. CDC tools tail that log:

DB ──writes──► Transaction Log ──tail──► CDC tool ──publish──► Kafka

Tool: Debezium (most common), Maxwell (MySQL), AWS DMS, GCP Datastream.

Why Tail the Log (Not Poll the Tables)

• Captures every change, even ones rolled back via undo.
• No load on application queries (log is decoupled).
• Captures ordering and transactions.
• Sub-second latency.

Polling SELECT * WHERE updated_at > ? is the alternative; it misses deleted rows and adds DB load.

Use Cases

Outbox vs CDC

Outbox:

• App writes events to an outbox table inside the business transaction.
• CDC (or a poller) streams the outbox.
• App controls event semantics ("OrderShipped" with rich payload).

Direct CDC of business tables:

• No outbox; CDC the orders table.
• Downstream sees raw rows; less business meaning.
• Schema changes (column adds/drops) directly affect downstream.

Most teams use outbox. Direct CDC is for systems where you control both ends and want zero app changes.

Schema Evolution

Tables change. CDC events must too:

• Add column: new events include the new field; old events don't (downstream tolerant).
• Drop column: old events still have it; downstream tolerant.
• Rename: best avoided; treat as drop+add.

Schema registry (Avro / Protobuf) on the Kafka side helps.

Initial Snapshot + Streaming

When you turn on CDC, you usually want both:

1. Snapshot: existing data (initial load).
2. Stream: future changes from the WAL.

Debezium does this automatically: takes a consistent snapshot using SELECT ... AS OF SCN (Oracle) or START TRANSACTION REPEATABLE READ (Postgres), then transitions to streaming from the recorded LSN.

Ordering

Per-table or per-key ordering preserved (CDC events flow through Kafka with key = primary key).

Across tables, ordering by transaction is best-effort. Multi-row transactions are visible as multiple events; downstream must reassemble if needed.

Consumer Considerations

CDC events are at-least-once. Consumers must dedup:

• Use Kafka offset + topic-partition for resume.
• Use event's unique LSN / position field for dedup.

For "rebuild downstream from scratch": consume from offset 0; idempotent application of events makes this safe.

Performance Impact on Source DB

CDC reads the WAL — but the WAL is already written for replication. Marginal extra load.

Caveats:

• Postgres: requires wal_level = logical and replication slots; abandoned slot fills disk.
• MySQL: row-based binlog format required.
• High write volume: CDC can produce huge Kafka traffic.

Schema Tools

Debezium emits events in a specific Avro / JSON Schema format. Schema Registry mandatory at scale.

{
  "schema": { "type": "record", ... },
  "payload": {
    "op": "u",
    "before": { ... },
    "after": { ... },
    "source": { "ts_ms": ..., "lsn": ... },
    "ts_ms": ...
  }
}

Failure Modes

Common Mistakes

[!NOTE] CDC is the canonical way to bridge ACID systems and event-driven downstream consumers. It's a different programming model — events instead of polled snapshots — but the payoff is massive: real-time integration with zero app code.

Interview Follow-ups

• "How would you migrate from a monolith DB to microservice DBs?" — Stand up microservice; CDC the monolith table; populate new DB; cut over; remove the old.
• "How does Debezium handle DDL?" — Captures DDL events too (where supported); downstream may need re-config.
• "How is CDC different from a trigger?" — Triggers are synchronous, in-DB, affect write latency. CDC reads the log post-commit — async, no app impact.

Q: Sidecar pattern and service mesh — what they solve.

Answer:

A sidecar is a helper process running alongside each app container, handling cross-cutting concerns (TLS, retries, telemetry) without app code changes. A service mesh is a fleet of sidecars + a control plane managing them centrally.

The Sidecar Pattern

Co-locate app + helper in the same pod / VM:

┌──────────────────────────────┐
│           Pod                 │
│ ┌─────────┐  ┌─────────────┐ │
│ │  App    │  │   Sidecar   │ │
│ │ (any    │◀▶│ (proxy /    │ │
│ │  lang)  │  │   utility)  │ │
│ └─────────┘  └─────────────┘ │
└──────────────────────────────┘

Communication: localhost (TCP / UDS) — fast, secure.

Examples:

• Logging agent: tails app stdout, ships to log aggregator.
• Metrics agent: scrapes app, pushes to TSDB.
• TLS terminator: handles HTTPS so app can be plain HTTP.
• Service mesh proxy: Envoy / Linkerd-proxy.

Service Mesh

                   ┌──────────────┐
                   │ Control Plane │  (Istiod, Linkerd control)
                   │ - config       │
                   │ - policy        │
                   │ - certs         │
                   └───────┬────────┘
                           │ pushes config
       ┌───────────────────┼────────────────────┐
       ▼                   ▼                    ▼
   ┌──────────┐       ┌──────────┐         ┌──────────┐
   │ App +    │       │ App +    │         │ App +    │
   │ Sidecar  │◄────► │ Sidecar  │◄──────► │ Sidecar  │
   └──────────┘       └──────────┘         └──────────┘

Each pod gets a proxy (Envoy in most). All traffic flows app → sidecar → network → sidecar → app.

Sidecars provide:

• mTLS between services (auto cert rotation).
• Retries, timeouts, circuit breaking.
• Load balancing (client-side).
• Per-request observability (metrics, traces).
• Traffic shifting (canary, blue/green, mirror).
• Authorization policies.

Popular Service Meshes

Why Sidecar Beats Library

Library approach: each service team integrates a resilience library.

• Per-language; need ports for Go, Python, Java, Node.
• Updates require app redeploys.
• Cross-language consistency hard.

Sidecar approach: one binary, language-agnostic.

• Update sidecar without touching app.
• Same behavior across all languages.
• Cost: extra process per pod.

Cost of Sidecars

Each pod runs an extra container:

• Memory: 50–200 MB per sidecar.
• CPU: 0.05 – 0.5 cores typical.
• Latency: ~1 ms added per hop (in + out).

For 1000-pod cluster: ~100 GB extra RAM, ~50 cores. Not trivial.

Alternatives

Library approach (no sidecar, in-process):

• Spring Cloud / Resilience4j.
• gRPC client interceptors.
• Per-language stack.

eBPF mesh (Cilium):

• No sidecar; kernel handles mTLS, load balancing.
• Lower overhead.
• Limited feature parity with Envoy.

Gateway-only (no per-pod proxy):

• mTLS at the cluster edge; plain inside.
• Cheaper; less defense-in-depth.

Sidecar Use Cases Beyond Mesh

• Logging: Fluent Bit collects app logs from disk → ships to ES/Loki.
• Secrets: Vault Agent renews secrets and mounts them as files.
• DB connection pooling: PgBouncer sidecar.
• Caching: Redis sidecar for in-pod cache.
• Compression / encryption: external for compute-heavy work.

Common Pitfalls

When NOT to Use a Mesh

• Small cluster (< 10 services): library / gateway is fine.
• Cost-sensitive (memory/CPU budget tight).
• All services are one language: native library may be lighter.
• No team to operate it (mesh demands SRE attention).