ProblemReal-time and messaging
Webhook delivery
Tell customers’ servers about events as they happen, safely, even when those servers are slow or down.
The brief
Webhooks platform · suggested for Senior
Design a webhook delivery system
Design the webhook service for a B2B platform: when something happens (an invoice is paid, a subscription changes), deliver the event to every customer endpoint that subscribed to it, in a way customers can trust, see and rely on, whatever their servers do.
You run the webhook layer of a payments-style platform. Internal services publish events, and each customer registers HTTPS endpoints on their own servers for the event types they care about. Customers build billing, provisioning and shipping on these events, so an accepted event must reach them, or they must see that it failed. Their servers are anything but reliable: some answer in milliseconds, some take seconds, some are down for hours, and some URLs are typed in wrong or on purpose. Customers see every attempt in a dashboard and can send failed events again. About 20,000 customers, 100,000 endpoints and 3 billion deliveries a day, in several regions. Design the service, from an internal service publishing an event to each customer endpoint’s reply.
Functional requirements
- CoreCustomers register HTTPS endpoints and choose which event types each one receives.
- CoreWhen an event happens (for example invoice.paid), it is delivered to every endpoint subscribed to its type.
- CoreCustomers see every delivery attempt for their endpoints, and can send one event again or every failed event in a time range.
- CoreCustomers can rotate an endpoint's secret without missing deliveries.
Non-functional requirements
- DurabilityEvery accepted event is delivered to each subscribed endpoint or, after retries for up to 3 days, marked failed where the customer can see it.
- SecurityReceivers can check that a delivery came from us, was not changed, and is not an old one sent again.
- LatencyA slow or failing endpoint must not delay deliveries to anyone else.
- LatencyThe first attempt reaches a healthy endpoint within 10 seconds of the event at the 95th percentile.
- SecurityEndpoint URLs are supplied by customers and must never be a way into our own network.
- AvailabilityAccepting events stays available (99.99%), even when a whole region is down.
- QualityEvents and delivery attempts are kept for 30 days.
Scale
- ScaleAbout 20,000 customers with about 100,000 endpoints.
- ScaleAbout 2 billion events a day, which become about 3 billion deliveries once each event goes to every endpoint subscribed to its type.
- ScaleThe busiest hour runs at about three times the average.
- ScaleEndpoint behaviour (this problem’s givens): most endpoints answer within about half a second; at any moment about 1% are slow (10 seconds or more) or down.
- ScaleThe largest customer receives about 10% of all deliveries.
- ScaleThe service runs in several regions.
Nice to mention (below the line)
- Below the lineTransforming payloads per customer.
- Below the lineBatching several events into one request.
- Below the lineReceiving inbound webhooks.
- Below the lineA no-code integrations UI.
Out of scope
- Out of scopeThe services that produce events.
- Out of scopeBilling customers.
- Out of scopeClient SDKs (we document how to verify a delivery).
The variant
One fully specified brief, with the level it suits best. You can practise it at any level.
- Webhooks platformSuggested for Senior
Deliver every event to every customer endpoint that asked for it: signed, retried for days, and never held up by someone else’s slow server.
9 steps · 7 must-pass · about 49 min at Senior
What each level is graded on
Webhooks platform, step by step. A result at a level needs its must-pass steps answered, and only the steps you answer count. Going up a level adds steps or holds them to a stricter bar.
| Step | Entry level | Mid-level | Your levelSenior | Staff |
|---|---|---|---|---|
| The numbers that shape the design (optional) | Graded | Graded | Graded | Graded |
| Events, endpoints and deliveries | Graded | Graded | Graded | Graded |
| What customers control, and what they can see | Graded | Must-pass | Must-pass | Must-pass |
| An event, end to end | Must-pass | Must-pass | Must-pass | Must-pass |
| Never lose an event | Must-passcan block | Must-passcan block | Must-passcan block | Must-passcan block |
| What the receiver can trust | Must-pass | Must-pass | Must-pass | Must-pass |
| An endpoint that keeps failing | — | Must-pass | Must-pass | Must-pass |
| One slow customer, everyone else fine | — | Must-pass | Must-pass | Must-pass |
| A URL that points inside | — | — | Must-pass | Must-pass |
| Big tenants, big backlogs, many regions | — | — | — | Must-pass |
| At this level | 6 steps 3 must-pass about 31 min | 8 steps 6 must-pass about 43 min | 9 steps 7 must-pass about 49 min | 10 steps 8 must-pass about 56 min |
Entry level6 steps · 3 must-pass · about 31 min
- The numbers that shape the design (optional)Graded
- Events, endpoints and deliveriesGraded
- What customers control, and what they can seeGraded
- An event, end to endMust-pass
- Never lose an eventMust-passcan block
- What the receiver can trustMust-pass
Mid-level8 steps · 6 must-pass · about 43 min
- The numbers that shape the design (optional)Graded
- Events, endpoints and deliveriesGraded
- What customers control, and what they can seeMust-pass
- An event, end to endMust-pass
- Never lose an eventMust-passcan block
- What the receiver can trustMust-pass
- An endpoint that keeps failingMust-pass
- One slow customer, everyone else fineMust-pass
SeniorYour level9 steps · 7 must-pass · about 49 min
- The numbers that shape the design (optional)Graded
- Events, endpoints and deliveriesGraded
- What customers control, and what they can seeMust-pass
- An event, end to endMust-pass
- Never lose an eventMust-passcan block
- What the receiver can trustMust-pass
- An endpoint that keeps failingMust-pass
- One slow customer, everyone else fineMust-pass
- A URL that points insideMust-pass
Staff10 steps · 8 must-pass · about 56 min
- The numbers that shape the design (optional)Graded
- Events, endpoints and deliveriesGraded
- What customers control, and what they can seeMust-pass
- An event, end to endMust-pass
- Never lose an eventMust-passcan block
- What the receiver can trustMust-pass
- An endpoint that keeps failingMust-pass
- One slow customer, everyone else fineMust-pass
- A URL that points insideMust-pass
- Big tenants, big backlogs, many regionsMust-pass
- Must-pass
- a result at that level needs this step answered.
- Can block
- a miss there is a Blocking issue at that level.
Going up a level
- Mid-level
- “What customers control, and what they can see” becomes must-pass.“What the receiver can trust” is graded more strictly.Adds “An endpoint that keeps failing”, must-pass.Adds “One slow customer, everyone else fine”, must-pass.
- Senior
- “An endpoint that keeps failing” is graded more strictly.“One slow customer, everyone else fine” is graded more strictly.Adds “A URL that points inside”, must-pass.
- Staff
- “A URL that points inside” is graded more strictly.Adds “Big tenants, big backlogs, many regions”, must-pass.
Skills this problem practises
Your coach tracks readiness per skill across every problem you practise. Each links to the lesson that teaches it.
- Capacity estimation
- API design
- Idempotency and retries
- Data modelling
- Choosing storage
- High-level architecture
- Queues and async messaging
- Traffic surges and backpressure
- Consistency and concurrency
- Replication and durability
- Failure modes and recovery
- Observability and operations
- Rate limiting and abuse
- Security and privacy
Worked solution
Try it first: practise a variant, then read the walkthrough and compare it with your own design. Reading it first makes the practice much less useful.
Learn: Design a webhook delivery systemYour history
Sign in to see your sessions on this problem.