Design a Distributed Message Queue

Design a queue and you are really designing a distributed append-only log, which is what Kafka is. The insight that makes the whole thing click: do not delete messages when they are consumed — keep the log and let each consumer track its own position.

Requirements

Functional

• Producers publish to a topic; consumers subscribe.
• Multiple independent consumer groups read the same topic.
• Messages are retained for a configurable period and can be replayed.
• Ordering guarantees, at least within a partition.

Non-functional

• High throughput — hundreds of thousands of messages/second.
• Durable: an acknowledged message survives broker failure.
• Horizontally scalable, low latency.

The core structure

A topic is split into partitions; each partition is an append-only log of messages with monotonically increasing offsets.

Everything important follows from this picture:

• Appends are sequential writes, which even spinning disks do fast. Kafka's throughput

comes from doing the disk's favourite operation.

• Consumers store an offset, not the broker. The broker does not track per-message

acknowledgements, so it holds almost no per-consumer state — which is why one broker can serve thousands of consumers.

• Replay is free. Reset the offset and read history again. That single property is why

event-sourced architectures are built on logs rather than traditional queues.

• Partitions are the unit of parallelism — and of ordering.

Ordering

Order is guaranteed within a partition, not across a topic. This is the trade-off

people miss, and it is the most likely follow-up.

To keep related messages ordered, partition by a key: hash(key) % partition_count. All events for one order id, or one user, land in the same partition and are therefore ordered relative to each other. Global ordering requires a single partition — which caps throughput at one machine, and is almost never the right choice.

Note the operational trap: changing the partition count rehashes keys, so a key that used to map to partition 2 now maps to 5, and its ordering guarantee is broken across the change. Over-provision partitions from the start.

Consumer groups

Each partition is assigned to exactly one consumer within a group.

6 partitions, 3 consumers  → 2 partitions each
6 partitions, 6 consumers  → 1 each, maximum parallelism
6 partitions, 8 consumers  → 2 consumers idle

So the partition count is the ceiling on consumer parallelism. Different groups read independently at their own offsets, which is how one event stream feeds analytics, search indexing and notifications simultaneously.

When a consumer joins or dies, the group rebalances — partitions are reassigned, which briefly pauses consumption. Frequent rebalances are a common production problem and worth mentioning.

Delivery guarantees

The practical answer, and the one interviewers want: **at-least-once delivery plus idempotent consumers.** True end-to-end exactly-once requires the offset commit and the side effect to be in one transaction, which is only possible when the sink cooperates. Making the consumer idempotent — keyed upserts, deduplication on a message id — gets you the same observable behaviour far more cheaply.

Replication and durability

Each partition has a leader and followers. Producers write to the leader; followers replicate. Followers that are caught up form the in-sync replica set (ISR), and a write is acknowledged once the required number of ISR members have it.

acks=0    fire and forget — fastest, can lose data
acks=1    leader only — lost if the leader dies before replicating
acks=all  every in-sync replica — durable, slower

If the leader fails, a controller elects a new one from the ISR. Electing an out-of-sync replica ("unclean leader election") restores availability at the cost of losing acknowledged messages — a CP-versus-AP choice worth naming.

Storage and retention

Each partition is a series of segment files. Old segments are deleted wholesale by time or size, which is far cheaper than deleting individual messages. Log compaction is the alternative retention policy: keep only the latest message per key, turning the log into a replayable snapshot of current state.

The other throughput trick worth mentioning: because consumers read the same bytes the producer wrote, the broker can use zero-copy (sendfile) to move data from page cache to socket without passing through user space.

Follow-ups to expect

• Consumer lag — the gap between the log end and a consumer's offset; the single most

important metric to monitor.

• Poison messages — a message that always fails; retry with backoff, then move it to a

dead-letter topic.

• Backpressure — a slow consumer just lags; the log absorbs the burst, which is a major

advantage over push-based queues.

• Priority queues — logs do not support priority; use separate topics per priority.
• Delayed delivery — not native; use a scheduler or per-delay topics.
• Exactly-once end to end — where the transaction boundary actually is.

Track this problem on the System Design sheet.