Loading…

Understanding the problem

An LLM gateway sits between companies’ applications and several hosted model providers. Companies want one API, one bill, consistent policy controls and less work dealing with provider outages. The gateway earns its place only if it does the hard shared work safely: identify the company, apply its allowed models and region, stream the answer without extra waiting, keep each company inside limits and produce a bill that can be defended. The dangerous parts are all multi-tenant: an answer cache, a log store, a quota counter or a failover rule can leak or starve neighbours when it is global instead of company-scoped.

The practice brief already states the requirements. In the interview, restate them as promises the gateway makes to each company, then separate what is in the request path from what can run later.

Functional requirements

CoreSend a conversation and settings to a named model, or to a route the company defined, and receive the answer as it is written.
CoreCompany admins choose which models and substitutes may serve them, their limits and budget, what may be stored, and their region.
CoreSee usage and cost per model, and receive an invoice that matches it.
Below the lineBatch and background jobs.They reuse routing, capacity and billing, but the interview is about the interactive streaming path.
Below the lineEmbedding and image endpoints.The same policy and ledger ideas apply; the API details are separate.
Below the lineA playground for trying prompts.It is a client of the gateway, not the core gateway.

Non-functional requirements

SecurityA company never receives another company’s prompts, answers or usage.
ConsistencyUsage is counted once, never lost or doubled, and billed to the right company, including answers that were cut short or retried.
AvailabilityNo company can exceed its limits or budget, and one company’s traffic can’t slow down the others.
LatencyThe gateway adds at most about 50 ms before the first token (p99), plus any checks a company turned on, and passes the answer on as the provider writes it.
AvailabilityThe gateway stays 99.95% available with one provider down, for every request that has a healthy model the company allowed in its region; answers come only from models the company allowed.

Scale

From the briefNumbers here are given in the practice brief.

ScaleAbout 500 companies; about 50M requests a day, peaking around 3,000 a second.
ScaleA typical request is about 1,500 prompt tokens and 300 output tokens; some prompts run to tens of thousands.
ScaleMost requests stream, and an answer takes several seconds.
ScaleThree or four providers; each account has limits on tokens and on requests in each minute.

Out of scope

Out of scopeTraining or hosting models ourselves.
Out of scopeThe providers’ internals.
Out of scopeThe companies’ own apps.

Check yourself

Why is a shared answer cache more dangerous than a shared policy table?
Show answerHide answer
The policy table is keyed by company and contains rules. A shared answer cache contains generated content that may include private data or company-specific instructions, so a bad lookup can directly leak another company’s answer.

The numbers that shape the design

From the briefNumbers here are given in the practice brief.

Use the brief’s scale to find the bottlenecks. The gateway does not train models, but it still carries tokens, open streams, connection state, usage records and logs. The provider-capacity number matters most, because provider limits are outside the gateway fleet.

Back of the envelope from the brief

Quantity	Value	What it changes
Peak requests	About 3,000 a second	Size load balancers, gateway instances and usage-event intake for the peak, not the daily average.
Token flow	About 5.4M tokens a second	Compare with provider-account limits and company plans. Tokens are the scarce resource.
Open answers	Tens of thousands at once if answers take several seconds	Use streaming-friendly servers, timeouts and bounded buffers.
Usage events	Roughly one per provider call	A durable queue and ledger must handle peak writes without losing records.
Logs	Facts are much smaller and safer than full text	Default to metadata; make text retention an explicit company setting.
Provider capacity	Hundreds of millions of tokens a minute at peak	Plan accounts, regions and graceful degradation before launch traffic arrives.

What the numbers tell you

Requests per second size the gateway fleet, but tokens per minute size provider capacity and quotas.
Open streams size memory, sockets and deploy draining; they are not the same as completed requests.
Full prompt and answer logs are both larger and more sensitive than request facts, so storage choice is a product and privacy decision.

Architecture diagramThe numbers that shape the gateway

Read:Learn: Numbers and capacity estimation Learn: Latency budgeting

Try it: The numbers that shape the designGuided practice · AI inference gateway, Multi-tenant model gateway · Senior

Check yourself

Which number decides how many provider accounts or providers you need?
Show answerHide answer
Tokens per minute at peak, by route and region. Gateway instances can be added quickly; provider capacity is bounded by account limits and lead times.

Core entities

Keep the model centred on company policy and request evidence. The gateway needs to answer a request without a database round trip, so policy is cached near each gateway instance, with a durable store as source of truth. Request facts are written once the call is known, then usage events make them billable. Prompt and answer text are separate because most companies should not store them by default.

The records that matter

Entity	Key fields	Why it exists
Company	id, name, home region, plan	Owns policy, budgets and data controls.
API key	key hash, company id, scopes, status	Authenticates a request without storing raw keys.
Policy	allowed routes, substitutes, region, token limits, budget, logging mode, checks	The rule set every request must follow.
Model route	route id, provider, region, version, capabilities, price version	Maps the common API to a provider target and a billing rule.
Request record	request id, company, route, status, timings, policy version	A traceable fact that the gateway handled a request.
Provider call	call id, request id, provider account, status, usage source	The billable unit for retries, cut-off streams and reconciliation.
Usage ledger entry	call id, company, tokens, cost, price version	The invoice source, deduped by call id.
Log object	request id, storage mode, region, retention, key id	Only present when policy allows text or extra diagnostics.

Which store owns which job

Store	Use it for	Do not use it for
Postgres	Companies, keys, policies, route catalogue and ledger rows that need constraints	Hot per-token counters on every chunk.
Redis or quota service	Fast shared allowance checks and in-progress caps	Monthly invoices.
Kafka or durable queue	Usage events and reconciliation jobs that must survive restarts	Synchronous token streaming.
Object storage	Opt-in text logs in the company region	Global shared prompt storage.

Architecture diagramThe records the gateway needs

Read:Learn: Data model design Learn: PostgreSQL Learn: Apache Kafka Learn: S3 / Blob Storage

Try it: What the gateway storesGuided practice · AI inference gateway, Multi-tenant model gateway · Senior

Check yourself

Why keep provider call as a separate record from request?
Show answerHide answer
One client request can create more than one provider call when a retry runs again or a hedge is started. Billing and reconciliation need the provider-call identity.

The API

Expose one company-facing contract and hide provider-specific APIs behind the gateway. The main call accepts either a named model route or a company-defined route. The response is a stream of events, with a final event that names the served route and usage when the provider supplies it. Distinct errors matter because the caller should slow down for a limit, stop for budget and retry later when no allowed route is healthy.

Company-facing calls

Call	Request	Response	Errors
POST /v1/chat	route or model, messages, settings, max output, stream flag, idempotency key	Event stream of answer chunks, then final request id, served route, status and usage where available	400 bad input; 401 bad key; 402 budget; 429 limit; 503 no allowed healthy route
GET /v1/usage	date range, route filter, cursor	Usage and cost grouped by route and day, with links to request ids	401 or 403 for auth; 422 for invalid range
PUT /v1/policy	allowed routes, substitutes, limits, budget, logging, checks, region	Saved policy version	403 if not an admin key; 409 if a concurrent edit won
GET /v1/models	optional region	Routes and capabilities this company may use	401 for bad key
POST /v1/requests/{id}/cancel	request id or client idempotency key	Cancellation accepted, with final status later	404 unknown; 409 already finished

Architecture diagramOne streaming API for every provider

Diagram as text

Components: Company app (Client); POST /v1/chat (API gateway); Provider API (LLM); Final usage event (Analytics); Budget, limit, no model errors (Service). Connections: Company app → POST /v1/chat (conversation, route, stream); POST /v1/chat → Provider API (translated request); Provider API → Company app (token events); POST /v1/chat → Final usage event (request id and usage), asynchronous; POST /v1/chat → Budget, limit, no model errors (distinct errors), asynchronous.

Read:Learn: API contract design Learn: Protocols and real-time transports Learn: API Gateway

Try it: The APIGuided practice · AI inference gateway, Multi-tenant model gateway · Senior

Check yourself

Why not pass provider errors straight through?
Show answerHide answer
Companies call one gateway contract. They need stable meanings for budget, limit and no allowed route, not provider-specific error shapes that change by route.

High-level design

The high-level design is the request path, not a box diagram alone. The interviewer is listening for the order of checks, the latency budget before the provider call, streaming through and what happens when the caller leaves.

From request to first token

From the briefNumbers here are given in the practice brief.

A load balancer sends the request to any stateless gateway instance. Response buffering is disabled so chunks are not collected before the app sees them: the WHATWG HTML Standard defines server-sent events as a text/event-stream format for server-to-page messages, and it fits this one-way stream (WHATWG HTML Standard (opens in a new tab)).

If a proxy is in front, configure it for streaming. nginx, for example, says that when proxy buffering is disabled, the response is passed to the client immediately as it is received (nginx docs (opens in a new tab)).

The gateway checks the key, loads the policy from a local cache, reserves quota, optionally checks cache and guardrails, chooses a healthy allowed provider route and opens the provider stream. Each provider chunk is flushed through. If the caller disconnects or sends stop, the gateway sends cancellation to the provider, records what is known and lets billing or credits finish from the usage path.

The app calls the common streaming API with an API key and request id.
The gateway authenticates the key and reads the policy from local cache; a stale policy has a short versioned window.
Admission reserves tokens and spend, and optional checks run only if the company enabled them.
The router selects a healthy route from the company’s allowed list and region.
Provider chunks are written through to the caller; a slow reader gets a bounded buffer, not unbounded memory.
Disconnect or stop sends cancellation upstream, then finalises the request record with any provider work already produced.

Architecture diagramFrom request to first token

Read:Learn: High-level architecture Learn: Real-time delivery architecture Learn: Latency budgeting

Try it: From a request to the first tokenGuided practice · AI inference gateway, Multi-tenant model gateway · Senior

Keep reading+1 min

Timeouts for long streams

Use separate timeouts for connecting to a provider, waiting for the first token, idle gaps between chunks and total stream age. The first protects the gateway’s latency promise; the idle timeout catches a stuck stream; the total age lets deploys and clients avoid holding forgotten connections forever.

Backpressure from a slow reader

Each stream gets a bounded buffer. If the app reads too slowly and falls past that buffer, the gateway closes the client stream and sends cancellation to the provider. That contains memory per tenant and turns an overloaded client into that client’s failure, not a platform failure.

Design an LLM gateway

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

Understanding the problem

Functional requirements

Non-functional requirements

Scale

Out of scope

The numbers that shape the design

Core entities

The API

High-level design

From request to first token

Timeouts for long streams

Backpressure from a slow reader

Sign in with Google to read the rest

Practise this

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

01Understanding the problem

Functional requirements

Non-functional requirements

Scale

Out of scope

02The numbers that shape the design

03Core entities

04The API

05High-level design

From request to first token

Timeouts for long streams

Backpressure from a slow reader

Sign in with Google to read the rest

12Practise this

Understanding the problem

The numbers that shape the design

Core entities

The API

High-level design

Practise this