Loading…

Understanding the problem

This is the product around a large language model. People chat from web and phone apps and watch replies appear as they are written. Conversations follow them across devices, branch when edited and can be shared read-only, and an opt-in memory carries facts between chats. The models run on your own GPU servers, which can’t grow for weeks. The hard parts are what the model reads each turn, whose data a request may touch, a reply that outlives its connection, safety on a reply still being written, and the evening peak.

The brief gives the requirements. Restate them as promises to each person, then cut what the design doesn’t need.

Functional requirements

CoreChat with the assistant from web and phone apps, see each reply appear as it is written, and stop a reply.
CoreKeep every conversation and list them on every device; edit an earlier message or ask for another reply, with earlier versions still reachable.
CoreShare a read-only link to a conversation, and take it back.
CoreMemory (opt-in): the assistant remembers useful facts across conversations; people can see, correct and delete any of it, turn it off, or use a temporary chat that is kept only while it is open, never in their history or memory.
CorePlans: free users get a limited number of messages with the best model every few hours, then a smaller model; paid plans get more.
Below the lineUploading files and images.
Below the lineVoice conversations.
Below the lineSearching the web or using tools.
Below the lineProjects and folders of chats.
Below the lineSearching your own chat history.
Below the lineGroup conversations with several people.It changes the ownership rule; here each chat has one owner.
Below the lineExporting all of a person’s data.A batch job over the same per-user partitions.

Non-functional requirements

SecurityA person only ever sees their own conversations and what the assistant remembers about them, unless they share a link, and a link shows only what they chose to share.
QualityThe model reads a limited amount of text per request, its reply included. Conversations hundreds of messages long must keep working, and the assistant keeps following its own rules.
AvailabilityA reply being written survives a lost connection, a refresh or a switch of device, can be stopped, and is written and counted once.
SecurityNothing against the usage policy is shown to the user, even briefly, whether they ask directly or try to trick the model.
Security

Scale

From the briefNumbers here are given in the practice brief.

ScaleAbout 50M people a day, about 10 messages each; the evening peak is about three times the daily average.
ScaleReplies average about 500 tokens and take 10–30 seconds to write in full.
ScaleThe model reads at most about 32,000 tokens per request, its reply included (illustrative: real limits vary by model).
ScaleYour own GPU servers run the models: adding capacity takes weeks, and a server takes minutes to load a model.

Out of scope

Out of scopeTraining or fine-tuning models, and how the model servers work inside.
Out of scopeHow the safety classifier itself is built: assume one exists that answers in tens of milliseconds.
Out of scopeTools, web browsing and running code.
Out of scopeFiles, images and voice.
Out of scopePayments and billing.

Check yourself

Why is “the model remembers the conversation” the wrong starting point?
Show answerHide answer
The model keeps nothing between requests, so each turn the service decides what it reads, and a chat’s length becomes a limit and a cost.

The numbers that shape the design

Illustrative exampleNumbers here are illustrative: round figures to reason with, not measurements.

Use the brief’s scale to find what drives the design: the model fleet, the open streams and what each request carries.

Back-of-the-envelope numbers

Quantity	Value	Working
Messages a day	About 500M	50M people × about 10 messages.
Messages a second	About 5,800, or 17,000 at the peak	500M ÷ 86,400 seconds, then the brief’s 3× evening peak.
Replies being written at once	About 340,000	17,000 a second × about 20 seconds each: that many open streams.
Tokens written a second	About 8.5M at the peak	17,000 replies × about 500 tokens. This sizes the fleet.
Turns that fit the limit	About 50	(32,000 − 2,000 kept for the reply) ÷ an assumed 600 tokens a turn (a message and its reply).
Tokens sent to the model a day	About 1.5 trillion	500M × an assumed 3,000 tokens, growing every turn.
New text a day	About 1.25 TB	500M × an assumed 2.5 KB a message and its reply, before copies.

What the numbers tell you

The model fleet is the bottleneck, and it can’t grow tonight, so capacity is shaped rather than added; a long chat passes the limit after a few dozen turns.

Read:Learn: Numbers and capacity estimation Learn: LLM serving and cost

Try it: The numbers that shape the designGuided practice · ChatGPT-style assistant, Consumer chat assistant · Senior

Check yourself

Why multiply the peak rate by how long a reply takes?
Show answerHide answer
Each reply holds a stream and a server slot for its whole 10–30 seconds, so hundreds of thousands are open at once.

Core entities

A conversation is a tree of messages. Each message points to its parent, so an edit or another reply adds a sibling and the old version stays reachable; the conversation stores which leaf is current. Each reply records its status (writing, complete, stopped, failed or removed), the model version that wrote it and its token counts.

Partition conversations by user and messages by conversation, sorted by time, so “my chats, newest first” and “this chat, in order” are each one query. In DynamoDB an item’s key can’t be updated in place (AWS docs (opens in a new tab)), so last activity stays an ordinary attribute, listed through a global secondary index that DynamoDB updates automatically but asynchronously, so the list can trail a new message for a moment (AWS docs (opens in a new tab)).

What you store

Record	Key fields	Why it exists
Conversation	`conversation_id`, `user_id`, title, `current_leaf_id`, `is_temporary`, `updated_at`	The list on every device, and the current branch.
Message	`message_id`, `conversation_id`, `parent_id`, role, content, status, `model_version`, token counts	The tree: edits and other replies are siblings.
Memory	`memory_id`, `user_id`, text, `source_message_id`	What is remembered, and where it came from.
Shared link	`share_id` (random), `owner_id`, frozen copy, `revoked_at`	A read-only copy made at share time.
Plan usage	`user_id`, model tier, window, count	One shared counter for plan limits.

Architecture diagramMessages form a tree

Read:Learn: Data model design Learn: Partitioning, sharding and consistent hashing Learn: DynamoDB

Try it: What you storeGuided practice · ChatGPT-style assistant, Consumer chat assistant · Senior

Keep reading+1 min

Why one document per conversation won’t do

A long chat becomes one large document rewritten on every message, listing someone’s chats needs a scan, and a branch means copying. One row per message, sorted by time inside its conversation’s partition, makes appends cheap and both everyday reads a single query. Sharded Postgres by user works too; conversations idle for months move to cheaper storage.

Check yourself

The API

Sending is a POST that answers with an event stream: the reply id and the model, numbered pieces, then a final status. An edit is a new message under the same parent. The app makes up a client_message_id per message, so a retry returns the same reply. Past the best model’s allowance, the send still succeeds on the smaller model and says so; 429 is kept for too many replies at once.

Core calls

Call	Request	Response	Errors
`POST /v1/conversations/{id}/messages`	`client_message_id`, `parent_id`, content	Event stream: reply id and model, pieces, status	`401`, `404`, `429`
`GET /v1/messages/{id}/events?after=n`	Last piece the app has	The pieces after it, then live ones	`404`
`POST /v1/messages/{id}/stop`	None	Status `stopped`, text kept	`404`
`POST /v1/messages/{id}/regenerate`	`client_message_id`	A new sibling reply, streamed	`404`, `429`
`GET /v1/conversations?cursor=`	Cursor	Newest first, next cursor	`401`
`/v1/memories/{id}`, `/v1/conversations/{id}/share`	Read, edit, delete; share, revoke	The memory or link	`404`

Read:Learn: API contract design Learn: Idempotency and retries

Try it: The APIGuided practice · ChatGPT-style assistant, Consumer chat assistant · Senior

Check yourself

The network times out and the app sends again. What happens?
Show answerHide answer
The same client_message_id finds the existing reply, which streams from the start. One reply is written and counted.

High-level design

Draw one message end to end, then how the words reach the app.

One message, end to end

Number the hops. The message is saved before the model runs, the request goes to a server with room in the plan’s pool, and the reply streams back through the chat service, which saves each piece.

The app sends the message with its session token and a client_message_id.
The gateway checks the session and the plan, and passes the user id on.
The chat service saves the message, loads the branch and this turn’s memories, and builds the request.
The message is checked, and the router picks a server in the plan’s pool.
Tokens stream back, are saved as numbered pieces and reach the app.
At the end, usage is settled and a title and memory updates are queued.

Architecture diagramOne message, end to end

Who is asking comes from the session; the message is saved before the model runs; slower work comes after.

Diagram as text

Components: App (Client); Gateway (API gateway); Chat service (Service); Conversation store (NoSQL DB); Safety check (Service); Model router (Load balancer); Model server (LLM); Usage counter (Cache); Job queue (Queue). Connections: App → Gateway (message); Gateway → Chat service (user id); Chat service → Conversation store (save, load); Chat service → Safety check (check); Chat service → Model router (request); Model router → Model server; Chat service → Usage counter (settle), asynchronous; Chat service → Job queue (title, memory), asynchronous.

Read:Learn: High-level architecture Learn: Latency budgeting

Try it: One message, end to endGuided practice · ChatGPT-style assistant, Consumer chat assistant · Senior

Check yourself

Why save the message before calling the model?
Show answerHide answer
It exists under its id even if the model call fails, a retry finds it, and a second device shows it at once.

Getting the words to the app

The reply flows one way, so server-sent events over plain HTTP are enough, with an id on every piece (WHATWG HTML (opens in a new tab)). WebSockets suit two-way messages (RFC 6455 (opens in a new tab)); here stop and resume are separate calls.

The browser’s EventSource takes a URL and no body or custom headers, so it makes a plain GET, authenticated by cookie, and suits the events call, and it reconnects on its own with a Last-Event-ID header. The send is a POST, read as a streamed fetch response (Fetch Standard (opens in a new tab)). Fetch has no reconnect step like EventSource’s, so after a drop the app itself calls the events endpoint with its last number.

Turn response buffering off on every hop; nginx, for one, then passes the response on as it is received (nginx docs (opens in a new tab)). A comment line every 15 seconds or so keeps proxies from dropping idle streams, as the HTML standard suggests.

Architecture diagramGetting the words to the app

Design a ChatGPT-style assistant

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

Understanding the problem

Functional requirements

Non-functional requirements

Scale

Out of scope

The numbers that shape the design

Core entities

Why one document per conversation won’t do

The API

High-level design

One message, end to end

Getting the words to the app

Sign in with Google to read the rest

Practise this

Why not WebSockets everywhere?

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

01Understanding the problem

Functional requirements

Non-functional requirements

Scale

Out of scope

02The numbers that shape the design

03Core entities

Why one document per conversation won’t do

04The API

05High-level design

One message, end to end

Getting the words to the app

Sign in with Google to read the rest

12Practise this

Why not WebSockets everywhere?

Understanding the problem

The numbers that shape the design

Core entities

The API

High-level design

Practise this