ProblemData and infrastructure
Distributed message queue
A queue that many services send messages to and read them from, so work is handed over reliably.
The brief
Company-wide message platform · suggested for Senior
Design a distributed message queue
Design the company’s shared message platform: teams publish events to topics, and any number of reader groups read them at their own pace, with every acknowledged message surviving the loss of a broker or zone, order kept per key, and no team or slow reader able to hold up the rest.
You run the messaging platform for a large company. About 500 teams publish events (orders placed, payments taken, pages clicked) to topics, and other teams read them: billing, analytics, search, fraud. Many services rely on these events to move money and ship orders, so a message the platform has accepted must not be lost. Readers vary: some keep up within milliseconds, some run hourly batch jobs, and some stop for hours and come back. Some teams use a topic as a work queue shared by many workers; others replay a week of history to rebuild a database. The platform runs across three availability zones in one region, with a second region for disaster recovery. Design it, from a producer publishing a message to every reader group reading it.
Functional requirements
- CoreTeams create topics and choose how long messages are kept, or that only the latest message per key is kept.
- CoreProducers publish messages with a key; messages with the same key are read in the order they were published.
- CoreAny number of reader groups read each topic; each group reads every message, and within a group each message is handled by one reader.
- CoreGroups read at their own pace and can go back and read again within the retention period.
Non-functional requirements
- DurabilityA message acknowledged to a producer survives the loss of any one broker or availability zone.
- AvailabilityProducers can write again within about 10 seconds of a broker failure, and publishing is available 99.99% of the time.
- LatencyFor a reader that keeps up, a message is readable within 100 ms of being published at the 99th percentile.
- LatencyA slow or stopped reader group does not slow producers or any other group.
- QualityOne team’s traffic does not slow the other teams.
- AvailabilityThe platform keeps working from a second region if the main region is lost; losing a short, stated window of recent messages is acceptable then.
Scale
- ScaleAbout 500 teams and about 20,000 topics.
- ScaleAbout 1 million messages a second on average, averaging about 1 KB; the busiest hour runs at about three times the average.
- ScaleEach topic has one to twenty reader groups, about three on average.
- ScaleMessages are kept 7 days by default; some topics keep only the latest message per key, for ever.
- ScaleThree availability zones in one region, with a second region for disaster recovery.
Nice to mention (below the line)
- Below the lineFiltering or routing messages on the broker.
- Below the lineDelayed or scheduled messages.
- Below the linePriorities between messages.
Out of scope
- Out of scopeStream processing jobs.
- Out of scopeConnectors to databases and other systems.
- Out of scopeA schema registry.
The variant
One fully specified brief, with the level it suits best. You can practise it at any level.
- Company-wide message platformSuggested for Senior
One shared message platform every team publishes to and reads from at its own pace: an acknowledged message survives losing a broker or zone, order holds per key, and slow readers never slow writers.
9 steps · 7 must-pass · about 49 min at Senior
What each level is graded on
Company-wide message platform, step by step. A result at a level needs its must-pass steps answered, and only the steps you answer count. Going up a level adds steps or holds them to a stricter bar.
| Step | Entry level | Mid-level | Your levelSenior | Staff |
|---|---|---|---|---|
| The numbers that shape the design (optional) | Graded | Graded | Graded | Graded |
| Messages, storage and metadata | Graded | Graded | Must-pass | Must-pass |
| What producers and consumers call | Graded | Graded | Graded | Graded |
| A message, end to end | Must-pass | Must-pass | Must-pass | Must-pass |
| Never lose an acknowledged message | Must-passcan block | Must-passcan block | Must-passcan block | Must-passcan block |
| Consumers that crash, fall behind or choke | Must-pass | Must-pass | Must-pass | Must-pass |
| A broker dies | — | Must-pass | Must-pass | Must-pass |
| “Just make it exactly once” | — | Must-pass | Must-pass | Must-pass |
| One hot key | — | — | Must-pass | Must-pass |
| Many teams, more brokers, another region | — | — | — | Must-pass |
| At this level | 6 steps 3 must-pass about 31 min | 8 steps 5 must-pass about 43 min | 9 steps 7 must-pass about 49 min | 10 steps 8 must-pass about 56 min |
Entry level6 steps · 3 must-pass · about 31 min
- The numbers that shape the design (optional)Graded
- Messages, storage and metadataGraded
- What producers and consumers callGraded
- A message, end to endMust-pass
- Never lose an acknowledged messageMust-passcan block
- Consumers that crash, fall behind or chokeMust-pass
Mid-level8 steps · 5 must-pass · about 43 min
- The numbers that shape the design (optional)Graded
- Messages, storage and metadataGraded
- What producers and consumers callGraded
- A message, end to endMust-pass
- Never lose an acknowledged messageMust-passcan block
- Consumers that crash, fall behind or chokeMust-pass
- A broker diesMust-pass
- “Just make it exactly once”Must-pass
SeniorYour level9 steps · 7 must-pass · about 49 min
- The numbers that shape the design (optional)Graded
- Messages, storage and metadataMust-pass
- What producers and consumers callGraded
- A message, end to endMust-pass
- Never lose an acknowledged messageMust-passcan block
- Consumers that crash, fall behind or chokeMust-pass
- A broker diesMust-pass
- “Just make it exactly once”Must-pass
- One hot keyMust-pass
Staff10 steps · 8 must-pass · about 56 min
- The numbers that shape the design (optional)Graded
- Messages, storage and metadataMust-pass
- What producers and consumers callGraded
- A message, end to endMust-pass
- Never lose an acknowledged messageMust-passcan block
- Consumers that crash, fall behind or chokeMust-pass
- A broker diesMust-pass
- “Just make it exactly once”Must-pass
- One hot keyMust-pass
- Many teams, more brokers, another regionMust-pass
- Must-pass
- a result at that level needs this step answered.
- Can block
- a miss there is a Blocking issue at that level.
Going up a level
- Mid-level
- “Consumers that crash, fall behind or choke” is graded more strictly.Adds “A broker dies”, must-pass.Adds “Just make it exactly once”, must-pass.
- Senior
- “Messages, storage and metadata” becomes must-pass.“A broker dies” is graded more strictly.“Just make it exactly once” is graded more strictly.Adds “One hot key”, must-pass.
- Staff
- “One hot key” is graded more strictly.Adds “Many teams, more brokers, another region”, must-pass.
Skills this problem practises
Your coach tracks readiness per skill across every problem you practise. Each links to the lesson that teaches it.
- Capacity estimation
- API design
- Idempotency and retries
- Data modelling
- Choosing storage
- Sharding and partitioning
- High-level architecture
- Queues and async messaging
- Load balancing
- Traffic surges and backpressure
- Consistency and concurrency
- Replication and durability
- Failure modes and recovery
- Rate limiting and abuse
Worked solution
Try it first: practise a variant, then read the walkthrough and compare it with your own design. Reading it first makes the practice much less useful.
Learn: Design a distributed message queueYour history
Sign in to see your sessions on this problem.