Loading…

Understanding the problem

A sharing app takes hundreds of millions of uploads a day, and a small share break its rules. The design must be strict where the law is strict, fast for ordinary posters and fair to creators. People can look at well under 1% of uploads, so the questions are which checks run when, where the lines go, and what people spend their time on.

The practice brief states the requirements up front; confirm them in the first few minutes. The core is one strict rule about known illegal media, a pipeline that checks everything without slowing uploads, and decisions people can trust and challenge.

Functional requirements

CoreEvery upload is checked, and known illegal media is stopped before anyone can see it.
CoreEach upload gets a decision per kind of harm: leave it, act on it, or send it to a person.
CoreReviewers decide the hard cases, and every decision is enforced on the item and the account.
CoreCreators are told why and can appeal; the app can explain any decision later.
Below the lineBuilding the classifiers themselves.Another team trains them; this design decides how their output is used, and how a new one is brought in safely.
Below the lineThe recommendation feed.
Below the lineTools used by law enforcement.
Below the lineLaw enforcement tools.The design keeps evidence and makes the reports the law requires; what happens after that belongs to others.

Non-functional requirements

LatencyAn upload returns within about 300 ms (p99).
ConsistencyKnown illegal media is never visible, not even briefly.
DurabilityNo upload is skipped, even when moderation falls behind.
LatencyThe most serious items are decided within about an hour.
ConsistencyEvery decision can be explained later: what decided it, under which rule, and why.
SecurityPeople who handle moderation see only what their job needs.
Below the lineClassifier decisions within seconds for every post.The brief asks for about a minute, which lets the classifiers batch work and wait out a peak, as long as nothing is skipped.

Scale

From the briefNumbers here are given in the practice brief.

ScaleAbout 200M uploads a day, peaking around 8,000 a second.
ScaleAbout 6 classifiers score each item.
ScaleAbout 2,000 reviewers, each deciding about 500 items a day.
ScaleAppeals answered within a few days.

Check yourself

Why is it acceptable for most posts to be visible before the classifiers have looked at them?
Show answerHide answer
The brief allows about a minute, and holding every post for every model would slow all uploads for a small share. The exceptions come first: known illegal media is checked before publishing, and risky uploads can be held.

The numbers that shape the design

Illustrative exampleNumbers here are illustrative: round figures to reason with, not measurements.

Work out only the numbers the design depends on, with rough figures. The one that shapes everything is how few uploads people can look at.

Back of the envelope at the peak

Quantity	Value	Working
Uploads	About 8,000 a second	From the brief; the daily average is 200M ÷ 86,400, about 2,300 a second.
Classifier calls	About 48,000 a second	About 6 classifiers per item, and more for the sampled frames of videos.
Items people can decide	About 1M a day	2,000 reviewers × 500 items each, from the brief.
Share of uploads people see	About 0.5%	1M of 200M a day; everything else is decided automatically.
Decision records	At least 36 TB a year	200M × 500 bytes × 365, with at least one decision per upload.

What the numbers tell you

The classifier load is several times the upload rate, which is why models run behind a queue.
People can see about one upload in two hundred, so the lines and the review order matter more than anything else.

Architecture diagramWhere the load lands

Diagram as text

Components: People posting (User); Upload service (Service); Classifiers (Worker); Decision records (SQL DB); Review queue (Queue); Reviewers (Client). Connections: People posting → Upload service (~8,000 uploads/s); Upload service → Classifiers (~48,000 calls/s); Classifiers → Decision records (36+ TB a year); Classifiers → Review queue (the unsure share); Review queue → Reviewers (~1M a day).

Read:Learn: Numbers and capacity estimation

Try it: The numbers that shape the designGuided practice · Content moderation, Photo and video sharing app · Senior

Check yourself

Why does the share of uploads people can see matter more than the upload rate?
Show answerHide answer
Machines can be added for the upload rate. People can’t be added in the same way, and with about 0.5% of uploads reaching a person, the design has to be sure the right 0.5% do.

Core entities: items, decisions and cases

The records answer one question: what happened to this item, who or what decided it, under which rule, and why. So the item points to its media, each model’s output keeps the model’s version, and every decision is a new row, not an edit. Cases, actions and account standing each have their own life, so each gets its own record.

The entities and their key fields

Entity	Key fields	Why it exists
Content item	`id`, uploader, type, `media_ref`, visibility, `created_at`	The thing being moderated; the media itself lives in object storage.
Model output	item, model, `model_version`, category, confidence	Traces a decision to the model that informed it.
Decision	item, category, action, rule and version, reason, decided by, `created_at`	One row per decision, not edited afterwards: the history.
Review case	item, category, status, reviewer, lease, `opened_at`	The work a person claims and decides.
Enforcement action	decision, target (item or account), action	What was done, and what a reversal undoes.
Account standing	account, strikes, status	Repeat offences lead to restrictions and suspension.

Architecture diagramWhat the system records

Read:Learn: Data model design Learn: Storage choice justification

Try it: What the system recordsGuided practice · Content moderation, Photo and video sharing app · Senior

Keep reading+1 min

Where the records live

A relational database fits decisions, cases and standing, because a decision, its action and the account’s strike change together; a wide-column store also works at higher write volume. Keep the media in object storage, referenced by id, so review tools fetch it only when a reviewer opens a case, and so deleting it is one operation.

Check yourself

Why not keep a moderation_status column on each post and update it?

The API

Each caller gets its own contract. The upload path asks whether a post may go live, and "not yet" is a real answer; product services read an item’s decision; reviewers claim and decide cases; creators appeal. Each has its own authorisation: service tokens, reviewer roles, or the user’s own sign-in.

The moderation calls

Call	Request	Response	Errors
`POST /v1/checks`	content id, media reference, uploader	`allow`, `hold` or `block`; safe to repeat for the same content id	`400`
`GET /v1/items/{id}/decision`	none	The current decision: action, category, reason	`404`
`POST /v1/review/tasks/claim`	queue, limit	Cases with a link to the media, each with a lease that expires	`403`
`POST /v1/review/tasks/{id}/decision`	lease id, action, rule, reason	The decision	`409` if the lease expired or the case was decided
`POST /v1/appeals`	decision id, statement	The appeal case	`404`

Read:Learn: API contract design Learn: Idempotency and retries

Try it: The APIGuided practice · Content moderation, Photo and video sharing app · Senior

Check yourself

Why does claiming review work use POST, not GET?
Show answerHide answer
Claiming changes state: it assigns cases to a reviewer for a while. A GET can be retried, prefetched or cached, which would hand out cases nobody meant to take.

High-level design

Build the design one path at a time. Each step adds only what that path needs; the deep dives then harden the parts an interviewer pushes on.

The upload: fast checks, then visible

The person posting waits only for the checks that can stop a post: the known illegal index, blocklists and uploader limits. The media goes to object storage, the post goes live (or is held, if the uploader is high risk), and an event for every upload goes onto a durable queue.

The app uploads the photo; the media goes to object storage.
The upload service checks the uploader’s limits, blocklists and the known illegal index.
A block stops the post here; otherwise it goes live, or is held if the uploader is high risk.
An event for every upload goes onto the moderation queue, kept until it is processed.

Architecture diagramThe upload: fast checks, then visible

The user waits only for checks that can stop a post: the known illegal index, blocklists and uploader limits. Every upload also goes onto a durable queue.

Diagram as text

Components: App (Client); Upload service (Service); Uploader limits (Rate limiter); Known illegal index (Search); Media storage (Object store); Visible to others (Service); Moderation queue (Queue). Connections: App → Upload service (post); Upload service → Uploader limits (within limits?); Upload service → Known illegal index (match?); Upload service → Visible to others (live, or held); Upload service → Media storage (store); Upload service → Moderation queue (every upload), asynchronous.

Read:Learn: High-level architecture Learn: Large file upload & blob handling

Try it: From upload to a decisionGuided practice · Content moderation, Photo and video sharing app · Senior

Check yourself

Why not run all six classifiers before the upload returns?
Show answerHide answer
Every post would wait for the slowest model, and video would multiply the work exactly at the peak. Only checks that must happen before anyone sees a post belong here.

From the queue to a decision

Illustrative exampleNumbers here are illustrative: round figures to reason with, not measurements.

Classifier workers read the queue and report a confidence for each kind of harm. Cheaper models go first and heavier ones run only when those are unsure; a video is checked on sampled keyframes and its audio transcript. A decision service compares the confidences with each category’s lines and allows the post, acts on it, or opens a review case. Under load the queue grows and a sweep finds anything still without a decision, so nothing is skipped.

Classifier workers read events from the queue.
Each kind of harm gets a confidence; heavier models run only when the cheaper ones are unsure.
The decision service compares them with the category’s lines.
It allows the post, acts on it, or opens a review case.

Architecture diagramFrom the queue to a decision

A reviewer decides

A borderline video becomes a review case with the classifiers’ output attached. A reviewer claims cases for a limited time, so two people don’t decide the same one, and decides each with a rule and a reason. Enforcement applies the action, records it, tells the creator which rule and why, and updates the account’s standing.

The decision service opens a case for the video.
A reviewer claims it with a lease and decides it with a rule and a reason.
Enforcement applies and records it, notifies the creator and updates the account.

Architecture diagramA reviewer decides

Design a content moderation system

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

Understanding the problem

Functional requirements

Non-functional requirements

Scale

The numbers that shape the design

Core entities: items, decisions and cases

Where the records live

The API

High-level design

The upload: fast checks, then visible

From the queue to a decision

A reviewer decides

Sign in with Google to read the rest

Practise this

Holding risky uploads

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

01Understanding the problem

Functional requirements

Non-functional requirements

Scale

02The numbers that shape the design

03Core entities: items, decisions and cases

Where the records live

04The API

05High-level design

The upload: fast checks, then visible

From the queue to a decision

A reviewer decides

Sign in with Google to read the rest

12Practise this

Holding risky uploads

Understanding the problem

The numbers that shape the design

Core entities: items, decisions and cases

The API

High-level design

Practise this