Loading…
Loading…
Design the inference layer for a B2B platform whose customers embed LLM features in their apps. Each tenant sends millions of prompt requests per day across multiple foundation-model providers. The gateway must route by tenant policy, cache responses safely, fall back across providers on outage, enforce per-tenant cost and rate budgets, stream tokens end-to-end, and keep an auditable log of every call. Cost runaway, provider outages, and noisy-neighbor tenants are the day-1 production failure modes.
Best after a few full reps. Expect follow-up questions, edge cases, and deeper trade-off discussion.
5 stages
50 min
Grade anytime
Workspace-first, hints visible, stage retry available. The cheap, repeatable loop — build the answer shape before you take it under pressure.
Solve once, compare against the checklist, then come back to the weak stage instead of starting over.
Strict timer, hints hidden, debrief deferred to the end. Use this once you can already structure a clean answer and want to pressure-test pacing and pushback.
Best after one structured rep · timed · focused on pacing and communication.
This is the framing pass. A strong answer quickly defines what the system must do, what quality bar it has to hit, and the numbers that will justify the rest of the design.
What must exist
What good looks like
Numbers to anchor the design
Each stage has a distinct job. Treat them like separate deliverables instead of one giant answer, and the round becomes much easier to navigate.
Define the contract clearly: the endpoints, auth boundary, error semantics, and the one or two decisions that matter most.
What you should produce
Define the API. How do customer applications send prompts? How does streaming work end-to-end? How do tenants override defaults like model pinning...
Strong answers cover
Lay out the main components and trace the write path, read path, and any async path cleanly.
What you should walk through
Walk me through the high-level architecture.
Strong answers cover
Pick the store, show the schema or key model, and explain why that storage choice fits the access pattern.
What you should lock in
Walk me through the data model.
Strong answers cover
Name the first bottleneck, failure modes, and the trade-offs that keep the system fast and reliable under pressure.
What you should pressure-test
Time for the deep dive. The most interesting failure modes here are provider outages cascading into the gateway, noisy-neighbor tenants starving ev...
Strong answers cover
Translate the prompt into concrete requirements, scale, and trade-offs before drawing architecture.
Give APIs in the API stage, data models in the storage stage, and failure modes in scaling. Don't blur them together.
Grade early, compare to the reference reasoning criteria, fix the biggest misses, and re-submit the weak stage instead of starting over.
Related topics