Design ChatGPT

The newest question on the sheet, and the one where generic answers fail hardest. The difference from every other system here: **the expensive resource is GPU time, not CPU, disk or bandwidth**, and a single request occupies it for seconds. Every design decision follows from that.

Requirements

Functional

• Multi-turn conversations with context retained across turns.
• Responses stream token by token rather than arriving at once.
• Conversation history, regeneration, multiple models.

Non-functional

• First token in under a second; the rest streams at reading speed.
• Handle a long, expensive request without blocking others.
• Degrade gracefully when GPU capacity is exhausted — it will be.

100M weekly users, 10M daily conversations
Average 10 turns, ~500 output tokens each
A GPU generates ~50-100 tokens/second for one request unbatched
⇒ capacity is measured in GPU-seconds, and it is the binding constraint

Architecture

Client ⇄ SSE ⇄ API gateway (auth, rate limits, quotas)
                    │
              request queue (priority by tier)
                    │
              inference scheduler ──→ GPU inference servers (batched)
                    │                       │ model weights resident in VRAM
                    │                       └ KV cache per active sequence
                    ├──→ conversation store (history, per user)
                    └──→ safety filters (input and output)

The queue between the API and the GPUs is not optional. GPU capacity is fixed and expensive, demand is spiky, and the alternative to queueing is failing requests.

Streaming

Waiting for a 500-token response before showing anything means five seconds of blank screen. Streaming makes the same latency feel fast.

Server-Sent Events — one-directional, plain HTTP, auto-reconnecting
  data: {"delta": "Hello"}
  data: {"delta": " there"}
  data: [DONE]

SSE rather than WebSockets, because the flow is one-directional and SSE works through ordinary HTTP infrastructure. The consequence to mention: **the connection is held open for the duration of generation**, so the gateway must handle many long-lived connections, and a client disconnect must cancel the generation rather than continuing to burn GPU time on output nobody will read.

Continuous batching

The single most important serving optimisation, and the thing that distinguishes a real answer.

GPUs are massively parallel: generating one token for one request wastes most of the hardware. Batching many requests together costs barely more than one. But requests have different lengths, and static batching — wait for the whole batch to finish — means the whole batch runs at the speed of its longest member.

Static:      [A····][B··········][C···]   idle GPU while waiting for B
Continuous:  as soon as A finishes, a new request takes its slot

Continuous (in-flight) batching replaces finished sequences immediately, keeping the

batch full. It is the difference between mediocre and good GPU utilisation, and it is what vLLM and TensorRT-LLM are built around.

The KV cache

Generating token N re-reads the attention keys and values of tokens 1..N-1. Recomputing them each step would make generation quadratic; caching them makes it linear.

The cost is memory: the KV cache grows with context length × batch size and competes with the model weights for VRAM. **KV cache memory, not compute, is usually what limits batch size** — which is why long contexts are disproportionately expensive and why

PagedAttention (vLLM's technique of managing the cache in fixed pages like virtual

memory, avoiding fragmentation) matters.

Two consequences worth stating: prefix caching lets a shared system prompt be computed once across many requests, and a conversation's growing history makes every subsequent turn more expensive than the last.

Conversation context

Models have a bounded context window, so a long conversation cannot simply be replayed.

conversations  conversation_id, user_id, title, created_at
messages       conversation_id, seq, role, content, tokens, created_at

Strategies when history exceeds the window, in increasing sophistication: truncate the oldest turns; summarise the earlier conversation into a compact note; or retrieve only the relevant earlier turns by embedding similarity. Note that **truncation must not evict the system prompt**, and that any change to the prefix invalidates prefix caching.

Capacity management

Demand exceeds GPU supply, so the design must say what happens then:

• Tiered queues — paid users get priority; free users queue longer.
• Rate limits and quotas — per user and per organisation; see

rate limiter.

• Model routing — send simpler requests to a smaller, cheaper model.
• Backpressure — a visible queue position beats a timeout.
• Autoscaling — slow and expensive, since loading model weights into VRAM takes

minutes; keep warm capacity rather than scaling reactively.

Safety

Input and output both pass through classifiers. Output filtering must run on the stream, which is genuinely awkward: you are checking text that has already been partially sent. Practical answers are buffering a small window before emitting, and retracting the message if a later chunk trips a filter.

Follow-ups to expect

• Multi-tenancy isolation — one user's long request must not starve others; that is what

continuous batching plus fair scheduling provides.

• Retrieval augmentation — a vector search step before generation, which adds latency

before the first token.

• Tool use — generation pauses, an external call runs, generation resumes with the

result appended.

• Caching responses — exact-match caching for identical prompts; risky for personalised

context.

• Cost attribution — token accounting per user for billing and quotas.
• Evaluation and rollout — A/B testing model versions, with the ability to roll back.

Track this problem on the System Design sheet.