Loading…

Understanding the problem

This is the internal service that runs a company’s own language models on its own GPUs for every team. Chat products stream replies to people who are waiting; other teams send millions of background requests due within a day. A model writes one token at a time, replies run from a few tokens to thousands, and each request holds GPU memory that grows as it goes. The hard parts are scheduling, memory, long prompts, repeated beginnings, two kinds of work on one fleet, and servers that take minutes to start.

The brief gives the requirements. Restate them as promises to each team, then cut what the design doesn’t need.

Functional requirements

CoreTeams send a prompt to a chosen model and get the reply streamed back as it is written, or whole; they can cancel a request.
CoreTeams submit large batches of background requests with a deadline, check progress and collect the results.
CoreEach team’s usage is measured, and no team can take more than its share of the fleet.
Below the lineFine-tuning and serving custom versions of a model.
Below the lineImages, audio and other kinds of input.
Below the lineCalling outside model providers.
Below the lineSafety filtering of prompts and replies.
Below the lineConversations, memory and retrieval.The products around the model own them; each request arrives complete.
Below the lineChoosing a model for each request automatically.Teams name the model they want; the service runs it.

Non-functional requirements

LatencyA model writes one token at a time, and one request may need 10 tokens while another needs 2,000. Short requests must come back fast even while long ones are being written.
LatencyInteractive requests get their first token within about a second, then a steady stream without long pauses.
AvailabilityA GPU’s memory holds the model’s weights plus state for every token each request has read or written, so it limits how many requests a server can work on at once. Requests must never fail because a server took on too much.
LatencyBackground requests finish within 24 hours and never slow down people who are waiting.
AvailabilityA request that fails because a server dies is retried; a server taken out of service never cuts a reply off.
Cost

Scale

From the briefNumbers here are given in the practice brief.

ScaleAbout 1,000 interactive requests a second at the weekday peak, about a third of that at night; about 30 million background requests a day.
ScalePrompts average about 2,000 tokens, with documents up to about 30,000; replies average about 300 tokens, from a few to about 2,000, at about 30 tokens a second.
ScaleEach GPU has about 80 GB of memory; the large model’s weights take about 40 GB; each token a request has read or written holds about 0.5 MB (illustrative).
ScaleAbout 400 GPUs; a new server takes about eight minutes to start and load a model.

Out of scope

Out of scopeTraining or fine-tuning models, and how a model works inside.
Out of scopeProxying to outside model providers.
Out of scopeBilling between teams.

Check yourself

Why can’t requests a second alone size this fleet?
Show answerHide answer
Each reply takes seconds and holds GPU memory throughout, so what matters is how many are in flight and what each holds.

The numbers that shape the design

Illustrative exampleNumbers here are illustrative: round figures to reason with, not measurements.

Use the brief’s scale to find what drives the design: how many replies are being written at once, what each holds in GPU memory, and so how many GPUs the peak needs.

Back-of-the-envelope numbers

Quantity	Value	Working
Replies being written at once	About 10,000 at the peak	1,000 requests a second × about 10 s each (a 300-token reply at about 30 tokens a second).
Memory one average request holds	About 1.15 GB	(2,000 prompt + 300 reply tokens) × about 0.5 MB a token.
Requests one GPU holds	About 35	About 40 GB left after the weights ÷ 1.15 GB.
GPUs at the peak, by memory	About 290	10,000 ÷ 35, before headroom.
GPUs at the peak, by output	About 290 again	1,000 × 300 = 300,000 tokens a second, ÷ about 1,050 a GPU (35 × 30).
Bulk, spread evenly	About 350 a second	30M a day ÷ 86,400 seconds; the deadline lets most of it wait for the night.
GPUs free at night	About 300	400 − about 97 (290 ÷ 3), with interactive traffic at about a third of the peak.
One 30,000-token document	About 15 GB	30,000 × 0.5 MB: over a third of one GPU’s room for requests.

What the numbers tell you

Memory and output speed both land near 290 GPUs, so size the fleet on the larger with headroom; the 400 GPUs cover the peak. Long documents are the memory hazard.

Architecture diagramOne GPU’s memory

Read:Learn: Numbers and capacity estimation Learn: LLM serving and cost

Try it: The numbers that shape the designGuided practice · LLM inference service, Internal inference platform · Senior

Check yourself

Why do the memory count and the output count agree here?
Show answerHide answer
They count the same work two ways: 35 requests at about 30 tokens a second is about 1,000 tokens a second per GPU. So they agree by construction. Measure what a GPU really writes; if it’s under that, the output count sizes the fleet.

Core entities and where the models run

Illustrative exampleNumbers here are illustrative: round figures to reason with, not measurements.

Three records tell the router who serves what: each model, a pool of servers per model size with a target number of copies, and each server with its state.

The large model’s weights take about 40 GB of an 80 GB GPU, leaving about 40 GB for requests, so each GPU holds one copy. Weights live in object storage and are copied ahead of time to each server’s local NVMe disk, so a restart loads from disk instead of downloading. A serving engine (vLLM, SGLang or TensorRT-LLM, for example) runs many requests together and manages each one’s memory.

What you store

Record	Key fields	Why it exists
Model	`model_id`, `weights_gb`, `gpus_per_copy`, `weights_uri`	Memory per copy, and where to load it.
Pool	`pool_id`, `model_id`, `target_copies`	One per model size; follows demand.
Server	`server_id`, `pool_id`, `gpus`, `state`	Where the router may send work.
Request	`request_id`, `team_id`, class, `max_tokens`, status, usage	A repeat finds the original.
Batch	`batch_id`, `team_id`, deadline, counts, results location	Progress against the deadline.

Architecture diagramWhere the models run

Read:Learn: Partitioning, sharding and consistent hashing Learn: LLM serving and cost Learn: S3 / Blob Storage

Try it: Where the models runGuided practice · LLM inference service, Internal inference platform · Senior

Keep reading+1 min

A model too big for one GPU

Split one copy across the GPUs of one machine. Tensor parallelism shares each layer’s work among them; pipeline parallelism gives each GPU a run of layers (vLLM docs (opens in a new tab)). Either leaves more memory per GPU for requests, but the GPUs exchange data at every step (tensor parallelism most, so it needs fast links), and the same docs warn of synchronisation overhead or added latency. Here the large model fits on one GPU, so one copy per GPU keeps things simple.

The API

The generate call carries the model, the prompt, max_tokens (required, so the server knows how far a reply can grow), stop sequences, the class (interactive or bulk) and a request_id the caller makes up, so a repeat returns the same request. A stream sends token events, then a final event with usage and why it stopped. Bulk jobs get their own calls, so nobody holds a million connections open.

Core calls

Call	Request	Response	Errors
`POST /v1/generate`	`model`, `prompt`, `max_tokens`, `stop`, `class`, `stream`, `request_id`	Token events, then usage and the stop reason	`400`, `401`, `429`, `503` with `Retry-After`
`DELETE /v1/generate/{request_id}`	None	Status `cancelled`, usage so far	`404`
`POST /v1/batches`	A file of requests, `model`, deadline	`202` and a `batch_id`	`400`
`GET /v1/batches/{batch_id}`	None	Status, done and failed counts, a results link	`404`

Read:Learn: API contract design Learn: Idempotency and retries Learn: Protocols and real-time transports

Try it: The APIGuided practice · LLM inference service, Internal inference platform · Senior

Check yourself

The person waiting closes the tab halfway through a reply. What should happen?
Show answerHide answer
The closed stream cancels the request; the server drops it at the next step and frees its memory, and usage covers the tokens written so far.

High-level design

Draw one streamed request end to end, then the way a million bulk requests come in.

One streamed request, end to end

Number the hops. The router decides the pool and the server; the scheduler inside that server decides when the request starts.

The team’s service calls the gateway, which checks the team, its allowance and max_tokens.
The router picks the pool for the model, then a server in it.
The server’s scheduler starts the request when there is room and runs it with the others.
Tokens stream back through the router as they are written.
At the end, usage is recorded for the team.

Architecture diagramOne streamed request, end to end

The router picks the server; that server’s scheduler decides when the request starts.

Diagram as text

Components: Team service (Client); Gateway (API gateway); Team allowance (Cache); Router (Load balancer); Scheduler on the server (Service); GPU (LLM); Usage per team (Analytics). Connections: Team service → Gateway (generate); Gateway → Team allowance (reserve); Gateway → Router (checked); Router → Scheduler on the server (pool, server); Scheduler on the server → GPU (when there is room); Scheduler on the server → Usage per team (at the end), asynchronous.

Read:Learn: High-level architecture Learn: Load balancing & traffic routing

Try it: One request, end to endGuided practice · LLM inference service, Internal inference platform · Senior

Check yourself

Where does a request wait, and who decides it may start?
Show answerHide answer
In its server’s queue. The server’s scheduler decides, because only it knows how much memory is free at this step.

A million requests that can wait

Bulk work comes in through its own door, so it can wait safely and run only when the pools have room.

The team uploads a file of requests with a deadline, gets 202 and a batch id, and each request becomes a queue entry.
A dispatcher releases entries as the pools report spare room.
Each result is stored under its request id, and the batch counts done and failed.

Architecture diagramA million requests that can wait

Design an LLM inference service

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

Understanding the problem

Functional requirements

Non-functional requirements

Scale

Out of scope

The numbers that shape the design

Core entities and where the models run

A model too big for one GPU

The API

High-level design

One streamed request, end to end

A million requests that can wait

Sign in with Google to read the rest

Practise this

What a lost machine takes with it

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

01Understanding the problem

Functional requirements

Non-functional requirements

Scale

Out of scope

02The numbers that shape the design

03Core entities and where the models run

A model too big for one GPU

04The API

05High-level design

One streamed request, end to end

A million requests that can wait

Sign in with Google to read the rest

12Practise this

What a lost machine takes with it

Understanding the problem

The numbers that shape the design

Core entities and where the models run

The API

High-level design

Practise this