Loading…

Understanding the problem

A B2B platform’s services produce events, such as an invoice being paid, and customers register HTTPS endpoints for the event types they want. The happy path is one POST per endpoint. The hard parts surround it: servers slow or down for days, receivers that must trust what arrives, customers who need to see what happened, and URLs typed in by strangers that the service calls from inside its network.

The practice brief sets the scope below. Confirm early that at-least-once delivery with a way to spot repeats is acceptable, and what customers expect about order.

Functional requirements

CoreCustomers register HTTPS endpoints and choose which event types each one receives.
CoreWhen an event happens (for example invoice.paid), it is delivered to every endpoint subscribed to its type.
CoreCustomers see every delivery attempt for their endpoints, and can send one event again or every failed event in a time range.
CoreCustomers can rotate an endpoint's secret without missing deliveries.
Below the lineTransforming payloads per customer.
Below the lineBatching several events into one request.
Below the lineReceiving inbound webhooks.The opposite direction, with its own problems of verification and buffering.
Below the lineA no-code integrations UI.A product on top of delivery, not part of it.

Non-functional requirements

DurabilityEvery accepted event is delivered to each subscribed endpoint or, after retries for up to 3 days, marked failed where the customer can see it.
SecurityReceivers can check that a delivery came from us, was not changed, and is not an old one sent again.
LatencyA slow or failing endpoint must not delay deliveries to anyone else.
LatencyThe first attempt reaches a healthy endpoint within 10 seconds of the event at the 95th percentile.
SecurityEndpoint URLs are supplied by customers and must never be a way into our own network.
Availability

Scale

From the briefNumbers here are given in the practice brief.

ScaleAbout 20,000 customers with about 100,000 endpoints.
ScaleAbout 2 billion events a day, which become about 3 billion deliveries once each event goes to every endpoint subscribed to its type.
ScaleThe busiest hour runs at about three times the average.
ScaleEndpoint behaviour (this problem’s givens): most endpoints answer within about half a second; at any moment about 1% are slow (10 seconds or more) or down.
ScaleThe largest customer receives about 10% of all deliveries.
ScaleThe service runs in several regions.

Out of scope

Out of scopeThe services that produce events.
Out of scopeBilling customers.
Out of scopeClient SDKs (we document how to verify a delivery).

Check yourself

Why rule out exactly-once delivery out loud?
Show answerHide answer
A request that times out may or may not have been processed. The honest design sends again (at-least-once) and gives receivers one id per event so a repeat is harmless.

The numbers that shape the design

Illustrative exampleNumbers here are illustrative: round figures to reason with, not measurements.

Turn the brief’s givens into rates, open connections and a backlog. Requests open at once equal the rate times how long each takes, and that shows what slow endpoints cost.

Back of the envelope from the brief

Quantity	Value	What it means
Deliveries	Brief: 3 billion a day; about 35,000 a second	3 billion divided by 86,400 seconds.
Busiest hour	About 105,000 a second	The brief’s three times the average.
Open to healthy endpoints	About 52,000 requests	99% of the peak at about half a second each.
Open to slow endpoints	About 10,500 requests	1% of the peak at 10 seconds each: a sixth of all open requests.
Largest customer, six hours down	About 75 million waiting	A tenth of 35,000 a second for 21,600 seconds.
Attempt records	About 3 TB a day; about 90 TB for 30 days	At 1 KB each, one attempt per delivery, before retries and copies.

Architecture diagramWhere the connections go at the busiest hour

Rate times duration (illustrative): 1% of the traffic holds a sixth of the open requests.

Diagram as text

Components: Producing services (Service); Webhook platform (Service); Healthy endpoints 99% (External API); Slow endpoints 1% (External API); Largest customer, 6 h down (Queue). Connections: Producing services → Webhook platform (about 105,000/s); Webhook platform → Healthy endpoints 99% (about 52,000 open); Webhook platform → Slow endpoints 1% (about 10,500 open); Webhook platform → Largest customer, 6 h down (about 75 million waiting).

Read:Learn: Numbers and capacity estimation

Try it: The numbers that shape the designGuided practice · Webhook delivery, Webhooks platform · Senior

Check yourself

Which number most shapes this design?
Show answerHide answer
The slow endpoints’ share of open connections: 1% of the traffic holds about a sixth of them. It forces scheduling per endpoint, limits on what each may hold and non-blocking senders.

The API

Illustrative exampleNumbers here are illustrative: round figures to reason with, not measurements.

Two audiences use it. Producing services make one call, publishing an event with their own key so a retried publish is not a second event. Customers register endpoints with the event types and payload version they expect, rotate secrets in place, and see and resend their deliveries without asking support.

A compact contract

Call	Request	Response	Errors
`POST /v1/events`	`Idempotency-Key`; `tenant_id`, `type`, `payload`	`202` with `event_id`; a repeat returns the same id	`400`; `413` too large; `429`
`POST /v1/endpoints`	`url`, `event_types`, `version`	`201` with `endpoint_id` and the secret, shown once	`400` not HTTPS or not public
`POST /v1/endpoints/{id}/rotate-secret`	Customer auth	The new secret; the old one stays valid for a while	`404`
`PATCH …/{id}`, `POST …/{id}/test`	Enable, disable, change event types; send a test	The endpoint; `202` for the test	`400`, `404`
`GET /v1/endpoints/{id}/deliveries`	`status`, `from`, `to`, `page_token`	Each delivery with every attempt: time, code, latency, excerpt	`404`
`POST /v1/deliveries/{delivery_id}/retry`; `POST /v1/endpoints/{endpoint_id}/resend`	Nothing; or `from`, `to`	`202`; a range becomes a paced job	`404`

Architecture diagram“We didn’t get it”, answered by the customer

Read:Learn: API contract design Learn: Observability & operations Learn: Idempotency and retries

Try it: What customers control, and what they can seeGuided practice · Webhook delivery, Webhooks platform · Senior

Keep reading+1 min

Making the publish key safe

The key and the event must be bound atomically. Either store the event and its (producer, key) row in one transaction and let a relay feed the log, or record the key with a fresh event id first and append under that id, re-appending on a retry that finds the key but no confirmed append. The matcher upserts deliveries by event id, so a double append is harmless. A key reused with different content is refused.

Proving the customer controls the URL

Many platforms send a one-time challenge the endpoint must echo before live events flow. It stops a customer pointing the platform at someone else’s server, and it catches a mistyped URL at registration instead of three days later.

High-level design

Two flows carry the design, both from the brief: an event for a customer with two healthy endpoints, and the same event when one endpoint takes 10 seconds to answer.

Publish, match, deliver

Billing only publishes. The events API writes the event and replies; a matcher finds, in a cache, the endpoints that want that type and writes one delivery for each. Stateless senders send each one on its own and record every attempt.

Billing calls POST /v1/events with its key and the event.
The API writes the event to the events log and replies 202 with the event id.
The matcher reads it and finds, in the subscriptions cache, the endpoints that want invoice.paid.
It writes one delivery each for A and B; senders post each and save every attempt.

Architecture diagramOne event, one delivery per endpoint

Billing only publishes; each endpoint gets its own delivery, sent on its own.

Diagram as text

Components: Billing service (Service); Events API (API gateway); Events log (Queue); Matcher (Service); Subscriptions cache (Cache); Endpoint A (External API); Senders (Worker); Deliveries queue (Queue); Endpoint B (External API). Connections: Billing service → Events API (publish); Events API → Events log (write, then 202); Events log → Matcher; Matcher → Subscriptions cache (lookup); Matcher → Deliveries queue (one for A, one for B); Deliveries queue → Senders (each on its own); Senders → Endpoint A (POST); Senders → Endpoint B (POST).

Read:Learn: High-level architecture Learn: Producer-consumer / work queue

Try it: An event, end to endGuided practice · Webhook delivery, Webhooks platform · Senior

Check yourself

Why not let billing post to each endpoint before it replies to its own caller?
Show answerHide answer
Every paid invoice would wait for the slowest customer’s server, and a customer’s outage would become a billing outage.

When one endpoint is slow

The same event, but endpoint B takes 10 seconds. Nothing upstream notices: billing already has its 202, and the matcher already wrote both deliveries. A’s sender gets its 2xx in a fraction of a second. B’s sender waits up to its timeout; if B fails, its delivery gets a later due time. A does not wait for B.

Read:Learn: Queues, delivery semantics and backpressure

Try it: An event, end to endGuided practice · Webhook delivery, Webhooks platform · Senior

Check yourself

What in this design is waiting for endpoint B?
Show answerHide answer
Only B’s own delivery and the one sender connection holding it. The deep dive on slow endpoints makes sure even that cannot spread.

Entity	Key fields	Why it exists
Event	`event_id`, `tenant_id`, `type`, payload by id, `created_at`	What happened; kept 30 days so failed deliveries can be sent again.
Endpoint	`endpoint_id`, `tenant_id`, `url`, `event_types`, `version`, secret reference, `status`	Where to send and what; the secret itself sits in a secrets manager.
Delivery	`endpoint_id`, hour, `created_at`, `event_id`; `status`, `attempts`, `next_attempt_at`	One per event and endpoint, read newest first for the customer’s log.
Delivery by event	`event_id`, `endpoint_id`	A copy that answers “what happened to event X” without a global index.
Attempt	delivery, `attempt_no`, `at`, `status_code`, `latency_ms`, reply excerpt	Every try, kept as its own row.

Design a webhook delivery system

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

Understanding the problem

Functional requirements

Non-functional requirements

Scale

Out of scope

The numbers that shape the design

Core entities

Where each record lives

The API

Making the publish key safe

Proving the customer controls the URL

High-level design

Publish, match, deliver

When one endpoint is slow

Sign in with Google to read the rest

Practise this

The answer in two minutes

Also asked as

What interviewers look for

The decisions that matter

01Understanding the problem

Functional requirements

Non-functional requirements

Scale

Out of scope

02The numbers that shape the design

03Core entities

Where each record lives

04The API

Making the publish key safe

Proving the customer controls the URL

05High-level design

Publish, match, deliver

When one endpoint is slow

Sign in with Google to read the rest

12Practise this

Understanding the problem

The numbers that shape the design

Core entities

The API

High-level design

Practise this