ProblemAI systems
LLM inference service
NewServe a company’s own language models to every team: fast streamed replies and millions of background jobs.
The brief
Internal inference platform · suggested for Senior
Design an LLM inference service
Design the internal service that runs your company’s own large language models on its own GPUs for every product team: chat products need replies that start within a second and stream steadily, and other teams send millions of background requests that must finish within a day.
Every product team calls one internal service to run the company’s language models; nobody outside the company uses it. A model writes its reply one token at a time, and lengths vary hugely: one request may need 10 tokens, another 2,000, and prompts range from a short question to a 30,000-token document. Each GPU holds a copy of the model’s weights, and for every request it is working on it also keeps state for every token read or written so far, which grows as the reply grows. Chat products stream replies to people who are waiting; other teams submit millions of background requests that must finish within a day. There are two model sizes. The GPU fleet is the company’s biggest computing cost, and a new GPU server takes minutes to start. Design the service.
Functional requirements
- CoreTeams send a prompt to a chosen model and get the reply streamed back as it is written, or whole; they can cancel a request.
- CoreTeams submit large batches of background requests with a deadline, check progress and collect the results.
- CoreEach team’s usage is measured, and no team can take more than its share of the fleet.
Non-functional requirements
- LatencyA model writes one token at a time, and one request may need 10 tokens while another needs 2,000. Short requests must come back fast even while long ones are being written.
- LatencyInteractive requests get their first token within about a second, then a steady stream without long pauses.
- AvailabilityA GPU’s memory holds the model’s weights plus state for every token each request has read or written, so it limits how many requests a server can work on at once. Requests must never fail because a server took on too much.
- LatencyBackground requests finish within 24 hours and never slow down people who are waiting.
- AvailabilityA request that fails because a server dies is retried; a server taken out of service never cuts a reply off.
- CostGPUs are the biggest cost: the fleet should not sit idle, and servers take minutes to start.
Scale
- ScaleAbout 1,000 interactive requests a second at the weekday peak, about a third of that at night; about 30 million background requests a day.
- ScalePrompts average about 2,000 tokens, with documents up to about 30,000; replies average about 300 tokens, from a few to about 2,000, at about 30 tokens a second.
- ScaleEach GPU has about 80 GB of memory; the large model’s weights take about 40 GB; each token a request has read or written holds about 0.5 MB (illustrative).
- ScaleAbout 400 GPUs; a new server takes about eight minutes to start and load a model.
Nice to mention (below the line)
- Below the lineFine-tuning and serving custom versions of a model.
- Below the lineImages, audio and other kinds of input.
- Below the lineCalling outside model providers.
- Below the lineSafety filtering of prompts and replies.
Out of scope
- Out of scopeTraining or fine-tuning models, and how a model works inside.
- Out of scopeProxying to outside model providers.
- Out of scopeBilling between teams.
The variant
One fully specified brief, with the level it suits best. You can practise it at any level.
- Internal inference platformSuggested for Senior
One internal service runs the company’s own language models on its GPUs for every team: streamed replies for people who are waiting, and millions of background requests a day.
9 steps · 5 must-pass · about 53 min at Senior
What each level is graded on
Internal inference platform, step by step. A result at a level needs its must-pass steps answered, and only the steps you answer count. Going up a level adds steps or holds them to a stricter bar.
| Step | Entry level | Mid-level | Your levelSenior | Staff |
|---|---|---|---|---|
| The numbers that shape the design (optional) | Graded | Graded | Graded | Graded |
| The API | Graded | Graded | Graded | Graded |
| One request, end to end | Graded | Graded | Graded | Graded |
| Where the models run | Graded | Graded | Graded | Graded |
| A ten-token answer stuck behind a long report | Must-pass | Must-pass | Must-pass | Must-pass |
| A 30,000-token contract meets a full GPU | Must-passcan block | Must-passcan block | Must-passcan block | Must-passcan block |
| Every contract makes everyone’s replies stutter | — | Must-pass | Must-pass | Must-pass |
| The same 9,000 tokens, read again and again | — | — | Must-pass | Must-pass |
| Monday 9 am: five million background jobs | — | — | Must-pass | Must-pass |
| Servers that take eight minutes to help | — | — | — | Must-pass |
| At this level | 6 steps 2 must-pass about 32 min | 7 steps 3 must-pass about 39 min | 9 steps 5 must-pass about 53 min | 10 steps 6 must-pass about 61 min |
Entry level6 steps · 2 must-pass · about 32 min
- The numbers that shape the design (optional)Graded
- The APIGraded
- One request, end to endGraded
- Where the models runGraded
- A ten-token answer stuck behind a long reportMust-pass
- A 30,000-token contract meets a full GPUMust-passcan block
Mid-level7 steps · 3 must-pass · about 39 min
- The numbers that shape the design (optional)Graded
- The APIGraded
- One request, end to endGraded
- Where the models runGraded
- A ten-token answer stuck behind a long reportMust-pass
- A 30,000-token contract meets a full GPUMust-passcan block
- Every contract makes everyone’s replies stutterMust-pass
SeniorYour level9 steps · 5 must-pass · about 53 min
- The numbers that shape the design (optional)Graded
- The APIGraded
- One request, end to endGraded
- Where the models runGraded
- A ten-token answer stuck behind a long reportMust-pass
- A 30,000-token contract meets a full GPUMust-passcan block
- Every contract makes everyone’s replies stutterMust-pass
- The same 9,000 tokens, read again and againMust-pass
- Monday 9 am: five million background jobsMust-pass
Staff10 steps · 6 must-pass · about 61 min
- The numbers that shape the design (optional)Graded
- The APIGraded
- One request, end to endGraded
- Where the models runGraded
- A ten-token answer stuck behind a long reportMust-pass
- A 30,000-token contract meets a full GPUMust-passcan block
- Every contract makes everyone’s replies stutterMust-pass
- The same 9,000 tokens, read again and againMust-pass
- Monday 9 am: five million background jobsMust-pass
- Servers that take eight minutes to helpMust-pass
- Must-pass
- a result at that level needs this step answered.
- Can block
- a miss there is a Blocking issue at that level.
Going up a level
- Mid-level
- Adds “Every contract makes everyone’s replies stutter”, must-pass.
- Senior
- Adds “The same 9,000 tokens, read again and again”, must-pass.Adds “Monday 9 am: five million background jobs”, must-pass.
- Staff
- Adds “Servers that take eight minutes to help”, must-pass.
Skills this problem practises
Your coach tracks readiness per skill across every problem you practise. Each links to the lesson that teaches it.
- Capacity estimation
- API design
- Sharding and partitioning
- High-level architecture
- Queues and async messaging
- Load balancing
- Traffic surges and backpressure
- Failure modes and recovery
- Observability and operations
- Latency budgets
- Rate limiting and abuse
- LLM serving and cost
Worked solution
Try it first: practise a variant, then read the walkthrough and compare it with your own design. Reading it first makes the practice much less useful.
Learn: Design an LLM inference serviceYour history
Sign in to see your sessions on this problem.