ProblemAI systems
ChatGPT-style assistant
NewA chat assistant for tens of millions: replies as they are written, every device, a memory people control.
The brief
Consumer chat assistant · suggested for Senior
Design a ChatGPT-style assistant
Design a chat assistant that tens of millions of people use every day from web and phone apps: replies appear as they are written, conversations follow people across devices, the assistant can remember things about them, and every reply stays within the usage policy.
People hold conversations with an assistant powered by large language models that your company runs on its own GPU servers. Each reply appears word by word and can take tens of seconds; people lose signal, refresh, stop replies and switch from phone to laptop mid-reply. Conversations are kept and listed on every device; people edit earlier messages, ask for another reply and share read-only links. With memory on, the assistant remembers useful facts about them across conversations, and they control what it keeps. Only the person can see their own conversations and memories. Free users get limited use of the best model; paying users must be served even at the evening peak, when the fleet can’t grow. Design the system.
Functional requirements
- CoreChat with the assistant from web and phone apps, see each reply appear as it is written, and stop a reply.
- CoreKeep every conversation and list them on every device; edit an earlier message or ask for another reply, with earlier versions still reachable.
- CoreShare a read-only link to a conversation, and take it back.
- CoreMemory (opt-in): the assistant remembers useful facts across conversations; people can see, correct and delete any of it, turn it off, or use a temporary chat that is kept only while it is open, never in their history or memory.
- CorePlans: free users get a limited number of messages with the best model every few hours, then a smaller model; paid plans get more.
Non-functional requirements
- SecurityA person only ever sees their own conversations and what the assistant remembers about them, unless they share a link, and a link shows only what they chose to share.
- QualityThe model reads a limited amount of text per request, its reply included. Conversations hundreds of messages long must keep working, and the assistant keeps following its own rules.
- AvailabilityA reply being written survives a lost connection, a refresh or a switch of device, can be stopped, and is written and counted once.
- SecurityNothing against the usage policy is shown to the user, even briefly, whether they ask directly or try to trick the model.
- SecurityA deleted memory or conversation stops shaping replies at once, and is gone everywhere within 30 days.
- LatencyThe first words of a reply appear within about a second, outside rare overloads.
- AvailabilityPaying users are still served at the evening peak, on a model fleet that can’t grow for weeks at a time.
Scale
- ScaleAbout 50M people a day, about 10 messages each; the evening peak is about three times the daily average.
- ScaleReplies average about 500 tokens and take 10–30 seconds to write in full.
- ScaleThe model reads at most about 32,000 tokens per request, its reply included (illustrative: real limits vary by model).
- ScaleYour own GPU servers run the models: adding capacity takes weeks, and a server takes minutes to load a model.
Nice to mention (below the line)
- Below the lineUploading files and images.
- Below the lineVoice conversations.
- Below the lineSearching the web or using tools.
- Below the lineProjects and folders of chats.
- Below the lineSearching your own chat history.
Out of scope
- Out of scopeTraining or fine-tuning models, and how the model servers work inside.
- Out of scopeHow the safety classifier itself is built: assume one exists that answers in tens of milliseconds.
- Out of scopeTools, web browsing and running code.
- Out of scopeFiles, images and voice.
- Out of scopePayments and billing.
The variant
One fully specified brief, with the level it suits best. You can practise it at any level.
- Consumer chat assistantSuggested for Senior
People chat with an assistant from web and phone apps, with replies that appear as they are written, on free and paid plans.
9 steps · 5 must-pass · about 53 min at Senior
What each level is graded on
Consumer chat assistant, step by step. A result at a level needs its must-pass steps answered, and only the steps you answer count. Going up a level adds steps or holds them to a stricter bar.
| Step | Entry level | Mid-level | Your levelSenior | Staff |
|---|---|---|---|---|
| The numbers that shape the design (optional) | Graded | Graded | Graded | Graded |
| What you store | Graded | Graded | Graded | Graded |
| The API | Graded | Graded | Graded | Graded |
| One message, end to end | Graded | Graded | Graded | Graded |
| A conversation that outgrows the model | Must-pass | Must-pass | Must-pass | Must-pass |
| Yours alone, until you share | Must-passcan block | Must-passcan block | Must-passcan block | Must-passcan block |
| A reply in a tunnel | — | Must-pass | Must-pass | Must-pass |
| A harmful answer, mid-stream | — | — | Must-pass | Must-pass |
| Twice the demand, the same servers | — | — | Must-pass | Must-pass |
| A better model, without a bad week | — | — | — | Must-pass |
| At this level | 6 steps 2 must-pass about 33 min | 7 steps 3 must-pass about 40 min | 9 steps 5 must-pass about 53 min | 10 steps 6 must-pass about 60 min |
Entry level6 steps · 2 must-pass · about 33 min
- The numbers that shape the design (optional)Graded
- What you storeGraded
- The APIGraded
- One message, end to endGraded
- A conversation that outgrows the modelMust-pass
- Yours alone, until you shareMust-passcan block
Mid-level7 steps · 3 must-pass · about 40 min
- The numbers that shape the design (optional)Graded
- What you storeGraded
- The APIGraded
- One message, end to endGraded
- A conversation that outgrows the modelMust-pass
- Yours alone, until you shareMust-passcan block
- A reply in a tunnelMust-pass
SeniorYour level9 steps · 5 must-pass · about 53 min
- The numbers that shape the design (optional)Graded
- What you storeGraded
- The APIGraded
- One message, end to endGraded
- A conversation that outgrows the modelMust-pass
- Yours alone, until you shareMust-passcan block
- A reply in a tunnelMust-pass
- A harmful answer, mid-streamMust-pass
- Twice the demand, the same serversMust-pass
Staff10 steps · 6 must-pass · about 60 min
- The numbers that shape the design (optional)Graded
- What you storeGraded
- The APIGraded
- One message, end to endGraded
- A conversation that outgrows the modelMust-pass
- Yours alone, until you shareMust-passcan block
- A reply in a tunnelMust-pass
- A harmful answer, mid-streamMust-pass
- Twice the demand, the same serversMust-pass
- A better model, without a bad weekMust-pass
- Must-pass
- a result at that level needs this step answered.
- Can block
- a miss there is a Blocking issue at that level.
Going up a level
- Mid-level
- Adds “A reply in a tunnel”, must-pass.
- Senior
- “Yours alone, until you share” is graded more strictly.Adds “A harmful answer, mid-stream”, must-pass.Adds “Twice the demand, the same servers”, must-pass.
- Staff
- Adds “A better model, without a bad week”, must-pass.
Skills this problem practises
Your coach tracks readiness per skill across every problem you practise. Each links to the lesson that teaches it.
- Capacity estimation
- API design
- Data modelling
- Choosing storage
- High-level architecture
- Queues and async messaging
- Networking and protocols
- Traffic surges and backpressure
- Failure modes and recovery
- Observability and operations
- Rate limiting and abuse
- Security and privacy
- Retrieval and RAG
- LLM serving and cost
- AI evaluation and guardrails
Worked solution
Try it first: practise a variant, then read the walkthrough and compare it with your own design. Reading it first makes the practice much less useful.
Learn: Design a ChatGPT-style assistantYour history
Sign in to see your sessions on this problem.