Loading…

Understanding the problem

This is an enterprise document assistant. Companies connect wikis, drives, tickets and PDFs. Employees ask natural-language questions and expect an answer written from their company’s documents, with links to the exact passages used. The hard part is not drawing a model box. It is keeping the answer path honest while documents, permissions and source systems keep changing. A correct design has two loops: an ingest loop that turns source documents into searchable passages, and an answer loop that searches only the passages this employee may use, then writes from them.

The brief gives the product requirements. Repeat them, then cut scope so the design can focus on answering and freshness.

Functional requirements

CoreConnect a company’s sources and keep what the assistant can use in step with them: new, edited and deleted documents, and changes to who may open them.
CoreAsk a question in plain language and get an answer written from the company’s documents, with links to the passages it used.
CoreRate an answer as helpful or not.
Below the lineFollow-up questions that build on earlier turns of a conversation.Useful, but a later layer can rewrite follow-ups into fresh search queries.
Below the lineUsage analytics for company admins.
Below the lineAnswers personalised to a person’s role or team.Permissions are mandatory; personalisation beyond them is policy on top.
Below the lineAdmin analytics and dashboards.Answer logs make them possible later.

Non-functional requirements

SecurityAn employee only ever gets answers, quotes and links from documents they are allowed to open in the source. No company ever sees another company’s content.
QualityWhen the documents don’t contain the answer, the assistant says so. An answer never states something its linked passages don’t support.
ConsistencyEdits and new documents show up in answers within about 10 minutes; deletions and removed access take effect within about 5 minutes.
LatencyThe first words of an answer appear within about 2 seconds at the weekday peak.
CostModel costs stay predictable as usage grows.

Scale

From the briefNumbers here are given in the practice brief.

ScaleAbout 5,000 companies and 10M employees.
ScaleAbout 50M documents in total, roughly 10 passages each; the largest company has about 2M documents.
ScaleAbout 5M document changes a day, plus changes to who may open them; a newly connected company can bring millions of documents at once.
ScaleAbout 20M questions a day, peaking around 500 a second on weekday mornings.
ScaleHosted models: the embedding model produces vectors of about 1,000 numbers; both models are billed per token and rate-limited.

Out of scope

Out of scopeTraining or fine-tuning models, and the models’ internals.
Out of scopeThe model providers’ data terms: assume what you send them isn’t used for training or kept, and can stay in the customer’s region.
Out of scopeBuilding the connectors: each source delivers documents, change events and access lists.
Out of scopeImages, audio and video.
Out of scopeTaking actions for the user, such as sending email or editing documents.
Out of scopeSearching the public web.

Check yourself

Why is fine-tuning out of scope for the core answer path?
Show answerHide answer
A fine-tuned model cannot reliably cite the current passage, forget a deleted document or respect per-document access at question time. The assistant needs retrieval at answer time.

The numbers that shape the design

Illustrative exampleNumbers here are illustrative: round figures to reason with, not measurements.

Use the brief’s scale to find design drivers: vector storage, embedding throughput, prompt size and concurrent streams.

Back-of-the-envelope numbers for the company knowledge assistant

Quantity	Value	What it means
Searchable passages	About 500M	50M documents times about 10 passages each.
Raw vectors	About 2 TB	500M passages times about 1,000 numbers times 4 bytes, before overhead.
Normal embedding work	Up to about 580 passages a second on average	5M changed documents a day times 10 passages divided by a day.
Full re-embed	A day or more	At an illustrative 5,000 passages a second, a little over a day.
Questions at peak	About 500 a second	Model capacity and streaming are explicit requirements.
Concurrent answers	About 5,000 streams	500 questions a second times an illustrative 10-second answer.

What the numbers tell you

The data volume pushes you toward sharded approximate vector search and a rebuildable index.
The full re-embed time turns a model change into a migration, not a deploy flag.
Prompt tokens and concurrent streams make retrieval quality a cost and latency lever.

Read:Learn: Numbers and capacity estimation Learn: Vector Database

Try it: The numbers that shape the designGuided practice · RAG assistant, Company knowledge assistant · Senior

Check yourself

Which number most changes the design?
Show answerHide answer
Passage count forces sharding, prompt tokens force retrieval discipline, and re-embed time drives the Staff migration plan.

Core entities

Store the original document and the searchable passages separately. The document store is the source of truth you can rebuild from. The passage index is a derived view optimised for retrieval. Lewis and coauthors frame RAG as combining generated answers with retrieved non-parametric memory, which is why the retrieved memory needs provenance and update paths (arXiv (opens in a new tab)).

For vector search, use an approximate nearest-neighbour index with filters rather than scanning every vector. HNSW is one common graph-based index with speed, recall and memory trade-offs (arXiv (opens in a new tab)). pgvector is a valid choice when you want Postgres semantics, but at this size you plan sharding and compression yourself; its README covers approximate search and quantization options (pgvector (opens in a new tab)).

Records to keep the answer path traceable

Record	Key fields	Why it exists
Company source	`company_id`, `source_id`, type, sync cursor	Connector state.
Document	`document_id`, `company_id`, source URL, title, version, access list, updated time	Store of record.
Passage	`passage_id`, `document_id`, version, position, text, vector, embedding version, access info	Search unit and citation target.
Answer	`answer_id`, user, question, cited passages, used documents, model route, created time	History, audit and cache tags.
Feedback	`answer_id`, helpful flag, comment	Production signals.

Passage lifecycle

Source change	Passage action	Important guard
New document	Parse, split, embed, write passages	Visible only after its version is complete.
Edit	Write new version, then retire old passages	Retries must not leave duplicate live versions.
Delete	Remove every passage by document id	Evict cached results.
Access change	Rewrite access info or check live	Revocation reaches retrieval.

Architecture diagramThe records that make answers traceable

Read:Learn: Data model design Learn: Indexes and search fundamentals Learn: Vector Database

Try it: What you storeGuided practice · RAG assistant, Company knowledge assistant · Senior

Keep reading+1 min

Filtering and pgvector at scale

Do not say pgvector is only for small systems. It can be the right default when relational semantics matter, but you own sharding, memory and compression. The pgvector changelog says version 0.8.0 added iterative index scans and better filtering cost estimation (changelog (opens in a new tab)), which is useful for access-aware retrieval.

Check yourself

The API

Keep the API small and asynchronous where work is slow. Questions stream text and citations. Source changes are accepted quickly and processed by workers. Deletes and sharing changes are first-class kinds.

Core calls

Call	Request	Response	Errors
`POST /v1/questions`	`question`, optional `conversation_id`	Event stream: text chunks, then `answer_id` and citations	`401`, `429`, `503` when the answer path is unavailable
`GET /v1/answers/{id}`	None	Stored answer and citations if still allowed	`404` if gone or no longer accessible
`POST /v1/sources/{id}/changes`	Batch of created, edited, deleted or re-shared documents	`202` with a `job_id`	`400`, `401`, `409` for an old cursor
`GET /v1/jobs/{id}`	None	Progress, failures and last processed cursor	`404`
`POST /v1/answers/{id}/feedback`	Helpful flag and optional comment	`204`	`404`

Read:Learn: API contract design Learn: Queues, delivery semantics and backpressure

Try it: The APIGuided practice · RAG assistant, Company knowledge assistant · Senior

Check yourself

Why return 202 for source changes instead of waiting for embedding?
Show answerHide answer
Parsing and embedding are slow and rate-limited. The connector needs durable acceptance and a job id, while workers process the change with retries.

High-level design

Draw delivery order: make source changes searchable, then answer one question with citations and access enforcement.

From a source to searchable

A connector gets durable acceptance. Workers fetch, parse, split, embed and write passages by document, version and position. Live edits and bulk imports use separate queues so imports do not starve daily changes.

The connector calls the changes API with document id, version, change kind and access information.
The API writes the event to the right queue and returns a job id.
Workers fetch the source file, parse it to text, split by structure and keep title and heading path with each passage.
Workers embed passages in batches with backoff, then write the new version and retire the old live version.
Repeated failures go to review; the source cursor advances only when the batch is durably recorded.

Architecture diagramFrom a source change to searchable passages

Live edits and bulk imports wait in priority queues, then idempotent workers update the store of record and the search index.

Diagram as text

Components: Wiki or drive (External API); Changes API (Service); Live and import queues (Queue); Parse and split (Worker); Embedding API (External API); Document store (Object store); Passage index (Vector DB). Connections: Wiki or drive → Changes API (change); Changes API → Live and import queues (accepted), asynchronous; Live and import queues → Parse and split (work); Parse and split → Embedding API (batches); Parse and split → Document store (parsed copy), asynchronous; Embedding API → Passage index (vectors).

Read:Learn: Queues, delivery semantics and backpressure Learn: Producer-consumer / work queue Learn: Vector Database

Try it: From a source to searchableGuided practice · RAG assistant, Company knowledge assistant · Senior

Check yourself

Why split a large first import from the live edit queue?
Show answerHide answer
The import can contain millions of files. If it shares one FIFO queue, an urgent edit or delete can wait behind bulk work and miss the freshness target.

From a question to an answer

From the briefNumbers here are given in the practice brief.

The question path retrieves first, then generates. The answer service reads identity from the verified session, builds the access filter, retrieves compatible candidates, reranks them, prompts with a few numbered passages and streams supported text. The brief’s first-words target keeps this path tight.

The app sends the question with the user’s normal auth token.
The answer service resolves company, user and current groups from trusted systems.
It runs keyword and vector retrieval inside the company and access filters, then reranks a candidate set.
It builds a prompt from the question, instructions and a small set of numbered passages.
The language model streams text; the service maps citations to document titles and links in the final event.
The answer log stores the used passages, source documents, route and feedback handle for audit and evaluation.

Architecture diagramFrom a question to the first words

Design a RAG assistant

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

Understanding the problem

Functional requirements

Non-functional requirements

Scale

Out of scope

The numbers that shape the design

Core entities

Filtering and pgvector at scale

The API

High-level design

From a source to searchable

From a question to an answer

Sign in with Google to read the rest

Practise this

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

01Understanding the problem

Functional requirements

Non-functional requirements

Scale

Out of scope

02The numbers that shape the design

03Core entities

Filtering and pgvector at scale

04The API

05High-level design

From a source to searchable

From a question to an answer

Sign in with Google to read the rest

12Practise this

Understanding the problem

The numbers that shape the design

Core entities

The API

High-level design

Practise this