ProblemData and infrastructure
Distributed key-value store
Store and fetch values by key across many machines, and keep answering when some of them fail.
The brief
Shared key-value store · suggested for Senior
Design a distributed key-value store
Design a durable, distributed key-value store that many product teams share: spread the keys over many machines, keep every acknowledged write safe through machine and zone failures, let teams choose how current a read must be, and keep working as the data grows and machines fail.
A company runs one shared key-value store for about 40 product teams. It holds shopping carts, user sessions and settings, device state, and records that must be claimed once, such as usernames. Each team has its own namespaces. The clients are the teams’ services, in the same region. The store holds about 30 TB of live data before any copies, about 30 billion keys, and grows about 50% a year. Values are about 1 KB typically and up to 1 MB; keys are up to 256 bytes. At peak it serves about 500,000 reads and 150,000 writes a second, and during a sale one team’s keys can take 10 times their usual writes. Machines have SSDs of about 8 TB. The company runs one region with three zones today, and a second region in Europe is planned within a year. Machines are restarted for patches every week, a disk or a machine dies about once a week, a zone lost power for 40 minutes once in two years, and a bad script once deleted a team’s whole namespace. Design the store: where each key lives, how a write is kept safe, what a read sees, and how the store keeps working through failures and growth.
Functional requirements
- CoreTeams get, put (with an optional expiry) and delete single keys in their own namespaces.
- CoreA team chooses, per request, between the fastest answer and one that includes every acknowledged write.
- CoreA team can write a key only if it doesn’t exist yet, or only if it hasn’t changed since it was read.
- CoreEach namespace has its own settings and limits.
Non-functional requirements
- DurabilityAt the default setting, an acknowledged write survives the loss of any one machine or any one zone.
- AvailabilityReads and writes keep working while one machine or one zone is down.
- ConsistencyA client that asks for it reads its own latest acknowledged write; weaker reads are allowed when a team chooses them.
- ConsistencyWhen two clients write one key at the same moment, the result is predictable, and the design says what is kept and what can be lost.
- LatencyA single-key get or put takes under 10 ms at p99 within a region, at the default setting.
- QualityAdding machines needs no downtime and moves only about the new machines’ share of the data.
- AvailabilityOnce the second region exists, a region outage loses at most a stated few seconds of writes, and the other region keeps serving.
Scale
- ScaleAbout 30 TB of live data before copies, about 30 billion keys, growing about 50% a year; values about 1 KB typically and up to 1 MB.
- ScaleAbout 500,000 reads and 150,000 writes a second at peak; one team’s keys can take 10 times their usual writes during a sale.
- ScaleMachines have SSDs of about 8 TB; they restart for patches weekly, and a disk or machine dies about once a week.
- ScaleOne region with three zones today (a zone lost power for 40 minutes once in two years); a second region, about 80 ms away, within a year.
Nice to mention (below the line)
- Below the lineRange scans and secondary indexes.
- Below the lineTransactions across several keys.
- Below the lineValues over 1 MB (kept in object storage).
- Below the lineA query language.
Out of scope
- Out of scopeThe teams’ own services and what they store.
- Out of scopeAuthentication internals: services already have identities.
- Out of scopeClient libraries for every language.
- Out of scopeHardware purchasing and billing.
The variant
One fully specified brief, with the level it suits best. You can practise it at any level.
- Shared key-value storeSuggested for Senior
One durable store for 40 teams: 30 TB over many machines, copies that survive a zone, honest reads, and a second region to come.
10 steps · 7 must-pass · about 48 min at Senior
What each level is graded on
Shared key-value store, step by step. A result at a level needs its must-pass steps answered, and only the steps you answer count. Going up a level adds steps or holds them to a stricter bar.
| Step | Entry level | Mid-level | Your levelSenior | Staff |
|---|---|---|---|---|
| The numbers that shape the design (optional) | Graded | Graded | Graded | Graded |
| What a stored value carries | Graded | Graded | Graded | Graded |
| The store’s API | Graded | Graded | Graded | Graded |
| Where a key lives | Must-pass | Must-pass | Must-pass | Must-pass |
| Copies that survive a zone | Must-passcan block | Must-passcan block | Must-passcan block | Must-passcan block |
| Inside one machine: from put to disk and back | Must-pass | Must-pass | Must-pass | Must-pass |
| A user saves a setting and reloads | Graded | Must-pass | Must-pass | Must-pass |
| Two writes to one cart, one split network | Graded | Must-pass | Must-pass | Must-pass |
| A machine restarts for ten minutes; another dies for good | — | Graded | Must-pass | Must-pass |
| From 30 machines to 45, during a sale | — | — | Must-pass | Must-pass |
| A second region, and a bad script | — | — | — | Must-pass |
| At this level | 8 steps 3 must-pass about 37 min | 9 steps 5 must-pass about 43 min | 10 steps 7 must-pass about 48 min | 11 steps 8 must-pass about 54 min |
Entry level8 steps · 3 must-pass · about 37 min
- The numbers that shape the design (optional)Graded
- What a stored value carriesGraded
- The store’s APIGraded
- Where a key livesMust-pass
- Copies that survive a zoneMust-passcan block
- Inside one machine: from put to disk and backMust-pass
- A user saves a setting and reloadsGraded
- Two writes to one cart, one split networkGraded
Mid-level9 steps · 5 must-pass · about 43 min
- The numbers that shape the design (optional)Graded
- What a stored value carriesGraded
- The store’s APIGraded
- Where a key livesMust-pass
- Copies that survive a zoneMust-passcan block
- Inside one machine: from put to disk and backMust-pass
- A user saves a setting and reloadsMust-pass
- Two writes to one cart, one split networkMust-pass
- A machine restarts for ten minutes; another dies for goodGraded
SeniorYour level10 steps · 7 must-pass · about 48 min
- The numbers that shape the design (optional)Graded
- What a stored value carriesGraded
- The store’s APIGraded
- Where a key livesMust-pass
- Copies that survive a zoneMust-passcan block
- Inside one machine: from put to disk and backMust-pass
- A user saves a setting and reloadsMust-pass
- Two writes to one cart, one split networkMust-pass
- A machine restarts for ten minutes; another dies for goodMust-pass
- From 30 machines to 45, during a saleMust-pass
Staff11 steps · 8 must-pass · about 54 min
- The numbers that shape the design (optional)Graded
- What a stored value carriesGraded
- The store’s APIGraded
- Where a key livesMust-pass
- Copies that survive a zoneMust-passcan block
- Inside one machine: from put to disk and backMust-pass
- A user saves a setting and reloadsMust-pass
- Two writes to one cart, one split networkMust-pass
- A machine restarts for ten minutes; another dies for goodMust-pass
- From 30 machines to 45, during a saleMust-pass
- A second region, and a bad scriptMust-pass
- Must-pass
- a result at that level needs this step answered.
- Can block
- a miss there is a Blocking issue at that level.
Going up a level
- Mid-level
- “A user saves a setting and reloads” becomes must-pass.“Two writes to one cart, one split network” becomes must-pass.Adds “A machine restarts for ten minutes; another dies for good”.
- Senior
- “A machine restarts for ten minutes; another dies for good” becomes must-pass.Adds “From 30 machines to 45, during a sale”, must-pass.
- Staff
- Adds “A second region, and a bad script”, must-pass.
Skills this problem practises
Your coach tracks readiness per skill across every problem you practise. Each links to the lesson that teaches it.
- Capacity estimation
- API design
- Data modelling
- Choosing storage
- Sharding and partitioning
- Consistency and concurrency
- Replication and durability
- Failure modes and recovery
- Observability and operations
Worked solution
Try it first: practise a variant, then read the walkthrough and compare it with your own design. Reading it first makes the practice much less useful.
Learn: Design a distributed key-value storeYour history
Sign in to see your sessions on this problem.