Module 13 · Cloud & System Design · Deep Dive

System Design Interviews

The 45-minute conversation that decides senior offers in the UAE market — a repeatable framework, the back-of-envelope math, the building blocks, and DocChat scaled to 100k users so the pattern sticks.

BasicIntermediateBuild

1 · The framework — drive the conversation

The single biggest mistake is to start drawing boxes in the first minute. Strong candidates spend the first five minutes narrowing the problem, then move through a predictable sequence. Memorise this order; it works for every prompt.

# The 8 steps — say them out loud, the interviewer is grading your process
1. Clarify requirements & scope   # what are we actually building? what's out of scope?
2. Back-of-envelope estimates    # QPS, storage, bandwidth — get the order of magnitude
3. Define the API                # the contract: a handful of endpoints
4. Data model                    # the core tables/entities and their keys
5. High-level architecture        # the boxes-and-arrows diagram
6. Deep-dive one component        # the interesting part — they'll pick or you offer
7. Identify bottlenecks           # where does this break at 10x? fix it
8. Discuss trade-offs             # every choice has a cost — name it

You won't finish all eight in depth — that's fine. The interviewer steers. Your job is to always know which step you're on and to think aloud. Silence reads as "stuck"; narration reads as "senior".

2 · Functional vs non-functional requirements

Step 1 splits into two buckets, and you must cover both. Juniors list features and stop. Seniors immediately ask about scale, latency, and consistency — the non-functional requirements that actually shape the architecture.

# Functional — WHAT the system does (features)
- A user uploads a PDF and asks questions about it
- The system answers using only that user's documents
- Users see their chat history

# Non-functional — HOW WELL it does it (the real design drivers)
- Scale:        how many users? docs per user? questions/day?
- Latency:      answer in under ~3s? upload feedback instant?
- Availability: 99.9%? can ingestion be eventually-done?
- Consistency:  must a just-uploaded doc be searchable immediately?
- Durability:   never lose an uploaded file
- Cost:         LLM calls are expensive — does that constrain us?

For DocChat, the killer non-functional questions are: "How many users, and how many documents each?" and "Is it OK if a freshly uploaded document takes a few seconds before it's searchable?". The answer to the second one (yes, it's OK) is what justifies an async ingestion pipeline — the centrepiece of the whole design.

3 · Back-of-envelope estimation

This is where most candidates freeze, and where you can shine with five memorised numbers. You don't need precision — you need the right order of magnitude, fast.

# Numbers to keep in your head (June 2026 mental model)
1 day        ~ 100,000 seconds   # (86,400 — round up, it's an estimate)
1 million    = 10^6
1 billion    = 10^9
1 char       ~ 1 byte
1 KB ~ 10^3  · 1 MB ~ 10^6 · 1 GB ~ 10^9 · 1 TB ~ 10^12

Now a worked estimate for DocChat. Assume 100k users, each holding 20 documents averaging 2 MB, and each user asks 5 questions/day.

# --- QPS (questions per second) ---
questions/day = 100,000 users * 5     = 500,000 / day
avg QPS       = 500,000 / 100,000 s   = 5 QPS
peak QPS      ~ 5 * 3 (peak factor)   ~ 15 QPS        # tiny! reads are cheap

# --- Storage (the raw files) ---
docs total    = 100,000 * 20          = 2,000,000 docs
file storage  = 2,000,000 * 2 MB      = 4,000,000 MB  = 4 TB
# trivial for object storage (S3). Grows ~linearly with users.

# --- Vector storage (the embeddings) ---
chunks/doc    ~ 40 chunks (a 2MB PDF ~ a few hundred KB text)
chunks total  = 2,000,000 docs * 40   = 80,000,000 chunks
per embedding = 1536 dims * 4 bytes    ~ 6 KB
vector data   = 80,000,000 * 6 KB     ~ 480 GB        # THIS is the number that matters

# --- Bandwidth / cost signal ---
each answer   ~ 1-2 LLM calls (embed query + generate)
LLM cost      = 500,000 answers/day * a few cents     # $$ — cost is a design driver

4 · Core building blocks

Almost every scalable web system is assembled from the same kit of parts. Know what each one buys you and what it costs — interviewers probe exactly here.

The mental model: a request hits a load balancer, lands on a stateless app server, which reads from a cache first and a database (often a read replica) on a miss. Anything slow — sending email, parsing a file, calling an LLM in bulk — gets pushed onto a queue and handled by separate workers, so the user's request returns fast.

5 · Consistency, availability & idempotency

The CAP theorem says that when the network partitions (some nodes can't talk to others), you must choose: stay Consistent (refuse stale answers) or stay Available (answer with possibly-stale data). You can't have both during a partition. In practice you reason per-feature, not per-system.

# Practical translation — pick per feature
Bank balance / payments  -> favour CONSISTENCY  (never double-charge)
Social feed / search      -> favour AVAILABILITY (a slightly stale feed is fine)
DocChat: doc is searchable a few seconds late -> AVAILABILITY is fine here

This is why DocChat can use eventual consistency for ingestion: the user uploads, gets an instant "processing…" response, and the document becomes searchable seconds later when the workers finish. We trade immediate consistency for a fast, always-available upload path — and that trade is the whole reason the pipeline is async.

Idempotency is the safety net that makes distributed systems survivable. An operation is idempotent if doing it twice has the same effect as doing it once. Queues deliver at least once, networks retry, users double-click — so your handlers will see duplicates.

# Make "ingest this document" idempotent with a stable key
def ingest(doc_id, content_hash):
    if already_processed(content_hash):   # dedupe on a stable key
        return                            # safe to call again — no double work
    chunks = chunk_and_embed(content)
    store(doc_id, chunks)
    mark_processed(content_hash)

6 · Rate limiting & designing for failure

At scale, everything fails — disks, networks, downstream APIs, your own bugs. Senior design is mostly about failing gracefully. Two themes interviewers love: protecting yourself with rate limits, and surviving downstream failures.

Rate limiting protects your system (and your budget) from abuse and runaway cost. The classic algorithm is the token bucket: each user gets tokens that refill over time; a request spends one; no tokens means a 429 Too Many Requests.

# Per-user token bucket, stored in Redis (shared across all app servers)
allowed = redis.call("rate_limit", user_id, limit=60, window="1m")
if not allowed:
    return http_429("Slow down — 60 questions per minute")

For surviving downstream failure (the LLM API, the database), three patterns work together:

# 1. Timeouts — never wait forever; fail fast
response = llm.generate(prompt, timeout=10)

# 2. Retries with backoff + jitter — for transient blips (NOT for bad input)
for attempt in range(3):
    try: return call()
    except Transient: sleep(2**attempt + random())  # jitter avoids thundering herd

# 3. Circuit breaker — stop hammering a dead dependency
if breaker.is_open():            # too many recent failures?
    return fallback()             # skip the call, degrade gracefully

Graceful degradation ties it together: if the primary LLM is down, fall back to a cheaper/secondary model; if vector search is slow, return a partial answer with a "still searching" note rather than an error page. A degraded answer beats a 500.

7 · Build: Design DocChat for 100k users

We've already done steps 1–2 (requirements above, and the math: tiny QPS, ~4 TB files, ~480 GB vectors, LLM cost is the constraint). Step 3, the API, is small and clean:

DocChat API (the contract)

POST /documents        # upload a file -> returns {doc_id, status: "processing"}
GET  /documents/:id    # poll status: processing | ready | failed
POST /chat             # {question} -> {answer, sources[]} (scoped to this user)
GET  /chat/history     # past conversations

Step 4, the data model, in managed Postgres:

users(id, org_id, email, ...)
documents(id, user_id, org_id, s3_key, status, content_hash)
chunks(id, document_id, user_id, org_id, text, embedding vector(1536))  # pgvector
messages(id, user_id, conversation_id, role, content, created_at)
# NOTE: every row carries user_id/org_id — that's our multi-tenancy guard rail

Step 5, the high-level architecture. The key idea: split the fast read path (asking questions) from the slow write path (ingesting documents) using a queue.

                         ┌──────────────┐
        users ──────────▶│ CDN / edge   │  static assets, signed file URLs
                         └──────┬───────┘
                                │
                         ┌──────▼───────┐
                         │ Load Balancer│
                         └──────┬───────┘
                                │
                ┌───────────────┼───────────────┐
                ▼               ▼               ▼
          ┌──────────┐   ┌──────────┐   ┌──────────┐   stateless
          │ App srv  │   │ App srv  │   │ App srv  │   (scale out)
          └────┬─────┘   └────┬─────┘   └────┬─────┘
               │              │              │
   ┌───────────┼──────────────┼──────────────┼─────────────┐
   ▼           ▼              ▼              ▼              ▼
┌──────┐  ┌─────────┐  ┌────────────────┐  ┌──────────┐  ┌──────────┐
│Redis │  │ Postgres│  │ Postgres read  │  │ Object   │  │ Queue    │
│cache │  │ primary │  │ replicas (RAG  │  │ store S3 │  │ (ingest) │
│+ rate│  │+pgvector│  │ vector search) │  │  files   │  └────┬─────┘
└──────┘  └─────────┘  └────────────────┘  └──────────┘       │
                                                         ┌─────▼──────┐
                                                         │  Workers   │
                                                         │ parse →     │
                                                         │ chunk →     │
                                                         │ embed →     │
                                                         │ store vecs  │
                                                         └─────┬──────┘
                                                               │ (LLM embedding API)
                                                               ▼
                                                        ┌─────────────┐
                                                        │ LLM provider│
                                                        │ (rate-limit)│
                                                        └─────────────┘

Now the deep-dives (step 6) — the parts the math told us are interesting:

file storage at scale

Never store the 4 TB of PDFs in the database. Put them in object storage (S3) — cheap, durable (11 nines), infinitely scalable. The DB holds only the s3_key pointer. Serve downloads via short-lived signed URLs through the CDN so files stay private but fast.

async ingestion queue + workers

Upload is the slow path: parse the PDF → split into chunks → call the embedding API → write 40 vectors. That can take many seconds and depends on a rate-limited LLM API — far too slow to block the user's request. So the app server just stores the file in S3, writes a documents row with status="processing", drops a message on the queue, and returns instantly. A pool of workers (scaled independently of app servers) drains the queue. Workers are idempotent (keyed on content_hash) because the queue is at-least-once. The user polls GET /documents/:id until it flips to ready.

vector search pgvector limits → when to shard

480 GB of vectors is the crux. pgvector on managed Postgres with an HNSW index comfortably handles millions of vectors and is the right starting point — one fewer system to operate, and search can target read replicas. But there's a ceiling: as vectors grow into the hundreds of millions, index build time, memory, and recall/latency degrade. Your scaling story, in order:

1. Start: pgvector + HNSW on Postgres            # simplest, good to ~10s of millions
2. Scale reads: replicas + tune index params      # cheap next step
3. Partition by tenant (org_id) or by user        # smaller indexes, natural isolation
4. Outgrow Postgres -> dedicated vector store      # when recall/latency/scale demand it

Crucially, partitioning by org_id doubles as multi-tenancy: each tenant's search only touches their own (smaller, faster) index, and there's no risk of leaking another tenant's chunks.

LLM cost & rate limits

The math flagged cost as the real constraint. Four levers, all interview gold:

1. Cache answers — identical/similar question -> serve cached result (Redis)
2. Batch — embed many chunks per API call during ingestion
3. Respect provider rate limits — token-bucket + queue smooths bursts
4. Fallback between models — primary down/throttled -> cheaper model (degrade)

multi-tenancy & observability

Isolation: every query is scoped by user_id/org_id at the data layer — never trust the client. This guarantees a user only ever retrieves their own chunks (a hard requirement; a leak here is a security incident). Observability: you can't operate what you can't see — emit structured logs, metrics (QPS, p95 latency, queue depth, LLM cost/spend, worker failures), and distributed traces. Queue depth climbing is your early warning that ingestion is falling behind.

Step 7–8, bottlenecks & trade-offs — say these out loud:

• Bottleneck: ingestion throughput. LLM rate limits cap how fast workers embed. Fix: more workers + batching + backpressure via the queue.
• Bottleneck: vector search at hundreds of millions. Fix: partition by tenant, then a dedicated vector store.
• Trade-off: eventual consistency on uploads. Cost: docs aren't instantly searchable. Benefit: fast, resilient upload path. Worth it for this product.
• Trade-off: pgvector now vs dedicated store later. Cost: a future migration. Benefit: ship faster with one less system. The right call early.

8 · Three classic prompts, same framework

The framework generalises. Here's the one sentence that defines each classic prompt's real challenge — practise running the full eight steps on each.

URL shortener

Easy to write, hard to scale reads. The crux: generating short, unique keys (base-62 of a counter or a hash) and a read-heavy redirect path that's basically a giant key→URL cache. Reads ≫ writes → lean on Redis + CDN; the DB is almost a backup.

Chat app

The crux is real-time delivery and presence: persistent connections (WebSockets) instead of request/response, fan-out of a message to all participants, and storing message history. Trade-off: at-least-once vs exactly-once delivery, ordering, and offline users (push notifications).

News feed

The classic fan-out on write vs read debate. Push (precompute each user's feed on post) gives fast reads but explodes for celebrities; pull (assemble at read time) is the reverse. Real systems do a hybrid. This is the prompt that teaches you "it depends — here's the trade-off," the most senior sentence in the room.

9 · Check yourself

Answer from memory — in an interview these come at you live, with no time to look anything up.