Loading…

Understanding the problem

Hundreds of teams publish events, such as orders placed and payments taken, to topics, and other teams read them: some within milliseconds, some hours later, some replaying a week. The happy path is an append and a read. The hard parts are keeping an acknowledged message through failures, keeping order per key while spreading gigabytes a second over many machines, and making sure no reader, team or hot key holds up the rest.

The practice brief sets the scope below. Settle early what an acknowledgement promises, which order readers need, and whether anyone needs exactly-once effects.

Functional requirements

CoreTeams create topics and choose how long messages are kept, or that only the latest message per key is kept.
CoreProducers publish messages with a key; messages with the same key are read in the order they were published.
CoreAny number of reader groups read each topic; each group reads every message, and within a group each message is handled by one reader.
CoreGroups read at their own pace and can go back and read again within the retention period.
Below the lineFiltering or routing messages on the broker.Readers filter what they read; a broker that only appends and serves ranges stays fast.
Below the lineDelayed or scheduled messages.A separate timer service can publish a message when it is due.
Below the linePriorities between messages.A log keeps arrival order; a topic per priority does the job.

Non-functional requirements

DurabilityA message acknowledged to a producer survives the loss of any one broker or availability zone.
AvailabilityProducers can write again within about 10 seconds of a broker failure, and publishing is available 99.99% of the time.
LatencyFor a reader that keeps up, a message is readable within 100 ms of being published at the 99th percentile.
LatencyA slow or stopped reader group does not slow producers or any other group.
QualityOne team’s traffic does not slow the other teams.
AvailabilityThe platform keeps working from a second region if the main region is lost; losing a short, stated window of recent messages is acceptable then.

Scale

From the briefNumbers here are given in the practice brief.

ScaleAbout 500 teams and about 20,000 topics.
ScaleAbout 1 million messages a second on average, averaging about 1 KB; the busiest hour runs at about three times the average.
ScaleEach topic has one to twenty reader groups, about three on average.
ScaleMessages are kept 7 days by default; some topics keep only the latest message per key, for ever.
ScaleThree availability zones in one region, with a second region for disaster recovery.

Out of scope

Out of scopeStream processing jobs.
Out of scopeConnectors to databases and other systems.
Out of scopeA schema registry.

Check yourself

Why ask what an acknowledgement promises before drawing anything?
Show answerHide answer
It decides the write path: copies in other zones before the acknowledgement cost a round trip per write, while acknowledging the leader’s own write is faster but can lose records.

The numbers that shape the design

Illustrative exampleNumbers here are illustrative: round figures to reason with, not measurements.

Turn the brief’s givens into bytes a second, disk and network, and storage. Copies multiply the writes, reader groups multiply the reads, and retention multiplies everything.

Back of the envelope from the brief

Quantity	Value	What it means
Data in	About 1 GB a second	Brief: 1 million messages a second at about 1 KB.
Busiest hour	About 3 GB a second	The brief’s three times the average.
Disk writes at the peak	About 9 GB a second	Every message stored three times.
Reads at the peak	About 9 GB a second	About three groups per topic, mostly recent data.
Seven days	About 605 TB; about 1.8 PB with copies	1 GB a second for 604,800 seconds, then three copies.
A group stopped six hours	About 1.1 TB behind	On a 50 MB a second topic, read back from disk.

Architecture diagramWhere the bytes go at the busiest hour

Illustrative peak figures from the brief: three copies triple the writes, and three reader groups triple the reads.

Diagram as text

Components: Producers (Service); Partition leaders (Queue); Two follower copies (Queue); About 3 groups per topic (Worker); Broker disks, 7 days (Component). Connections: Producers → Partition leaders (about 3 GB/s); Partition leaders → Two follower copies (about 6 GB/s); Partition leaders → About 3 groups per topic (about 9 GB/s); Partition leaders → Broker disks, 7 days (about 1.8 PB).

Read:Learn: Numbers and capacity estimation

Try it: The numbers that shape the designGuided practice · Distributed message queue, Company-wide message platform · Senior

Check yourself

Which number most shapes this design?
Show answerHide answer
The bytes: about 3 GB a second in becomes about 18 GB a second of disk and network with copies and readers, so the work must be sequential, spread out and served mostly from memory.

Core entities

Messages are records in an append-only log, one log per partition, stored as a series of segment files. Small indexes find a position, and retention deletes whole files. Cluster metadata is small but must agree everywhere; group positions change constantly, so they live apart from it.

The records that matter

Entity	Key fields	Why it exists
Record	`offset`, `timestamp`, `key`, `value`, `headers`	Written in compressed batches, not changed in place.
Segment	base offset, log file, sparse offset index, time index	Appends go to the active one; retention deletes whole closed ones.
Partition metadata	`topic`, `partition`, `leader`, `replicas`, in-sync set, leader epoch	Kept by the controller quorum and cached by every client.
Group position	`group`, `topic`, `partition`, next offset	Kept in an internal topic that holds only the latest value per key.
Topic	name, partitions, copies, retention or keep-latest, owning team	Set at creation, within the team’s limits.

Architecture diagramA partition on disk, and the metadata beside it

Read:Learn: Data model design Learn: Storage choice justification Learn: Write-heavy

Try it: Messages, storage and metadataGuided practice · Distributed message queue, Company-wide message platform · Senior

Keep reading+1 min

Why a log on plain files is fast

The work is sequential: appends at the end and reads from a position. Kafka writes to the filesystem without necessarily flushing, serves reads from the operating system’s page cache and, on plaintext connections, sends file data to the socket with sendfile, avoiding copies through the application; with TLS it cannot use sendfile (Kafka docs (opens in a new tab)). A sparse index, one entry every few kilobytes for example, is small enough to stay in memory.

The API

Illustrative exampleNumbers here are illustrative: round figures to reason with, not measurements.

Producers publish batches and get positions back; readers fetch from a position and wait a little when nothing is new; groups save their positions; and clients ask which broker leads each partition.

A compact contract

Call	Request	Response	Errors
`POST /v1/topics/{topic}/records`	Records with `key`, `value`, `headers`	Each record’s partition and offset	`not leader` (refresh, retry); `too many requests`; `record too large`
`GET /v1/topics/{topic}/partitions/{p}/records`	`offset`, `max_bytes`, `wait_ms`	Records and the latest offset; waits up to `wait_ms` if nothing is new	`offset out of range`; `not leader`
`POST /v1/groups/{group}/positions`	Next offset per partition	Saved	Refused if the member no longer owns the partition
`GET /v1/groups/{group}/positions`	None	Positions and latest offsets, so lag is visible	`404`
`POST /v1/topics`	Name, partitions, copies, retention or keep-latest	`201`	Over the team’s limits
`GET /v1/topics/{topic}/metadata`	None	Leader and replicas per partition	`404`

Read:Learn: API contract design

Try it: What producers and consumers callGuided practice · Distributed message queue, Company-wide message platform · Senior

Keep reading+1 min

Client defaults for 500 teams

In production this is a binary protocol over long-lived connections, with the same calls. Most teams keep the defaults, so choose them well: batch up to about 1 MB or a few milliseconds of linger, compress batches, let fetches wait for a minimum number of bytes up to a short limit, and retry with backoff. Batching is what makes gigabytes a second affordable within a 100 ms target, for example.

Check yourself

A producer gets “not leader”. What should it do?
Show answerHide answer

High-level design

Two flows from the brief carry the design: the orders service publishes order.created for order 42, and billing and analytics both read it, analytics an hour behind.

Publishing: key, partition, leader

Each topic is split into partitions spread over brokers, and each partition is an append-only log with one leader. The client library caches which broker leads each partition, hashes the key to choose one, and sends compressed batches to its leader.

The orders service hashes the key, order 42, to pick a partition, so every event for that order lands in the same one.
It looks up that partition’s leader in its cached metadata and sends a batch there.
The leader appends the batch to its log, and each record gets the next offset.
The leader answers with the offsets; the deep dive on durability says when that answer is safe.

Architecture diagramPublishing order 42

The client hashes the key, so every event for order 42 goes to the same partition and its leader.

Diagram as text

Components: Partition 0 leader (Queue); Orders service (Service); Partition 1 leader (Queue); Cached metadata (Cache); Partition 2 leader (Queue). Connections: Orders service → Partition 1 leader (order 42: a batch); Orders service → Partition 0 leader (other keys); Orders service → Partition 2 leader (other keys); Orders service → Cached metadata (who leads?).

Read:Learn: High-level architecture Learn: Apache Kafka

Try it: A message, end to endGuided practice · Distributed message queue, Company-wide message platform · Senior

Check yourself

Why hash the key rather than spread records round-robin?
Show answerHide answer
Records with the same key then land in one partition, which is the only place order holds. Spreading records evenly balances load but loses order per key.

Reading: every group at its own pace

Billing and analytics are separate reader groups, and each reads every record from its own offset. Billing gets new records from memory; analytics asks for records from an hour ago and reads them from disk. Records stay for 7 days whether read or not, and the broker does not wait for any reader.

Architecture diagramTwo groups read the same log at their own pace

Design a distributed message queue

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

Understanding the problem

Functional requirements

Non-functional requirements

Scale

Out of scope

The numbers that shape the design

Core entities

Why a log on plain files is fast

The API

Client defaults for 500 teams

High-level design

Publishing: key, partition, leader

Reading: every group at its own pace

Sign in with Google to read the rest

Practise this

Compaction and tombstones

Why not one queue that deletes on read

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

01Understanding the problem

Functional requirements

Non-functional requirements

Scale

Out of scope

02The numbers that shape the design

03Core entities

Why a log on plain files is fast

04The API

Client defaults for 500 teams

05High-level design

Publishing: key, partition, leader

Reading: every group at its own pace

Sign in with Google to read the rest

12Practise this

Compaction and tombstones

Why not one queue that deletes on read

Understanding the problem

The numbers that shape the design

Core entities

The API

High-level design

Practise this