ProblemData and infrastructure
Metrics and monitoring
Collect metrics from every server, keep months of history, and warn people within a minute of a problem.
The brief
Monitoring for a large company · suggested for Senior
Design a metrics and monitoring system
Design the metrics and alerting platform a large company runs for its own services: every service reports numbers such as request counts, errors and latency, engineers graph them on dashboards, and alert rules page the right on-call engineer when something is wrong.
The company runs about 2,000 services, owned by about 500 teams, across three regions. Every service reports numbers about itself, split by labels such as endpoint, status and region, and engineers graph them all day. Teams write alert rules that page their on-call engineer through the company’s paging tool. Today the system is a patchwork: some teams run their own, graphs are slow during incidents, and on-call engineers complain that they are paged too often for things that don’t matter. Design one platform for everyone.
Functional requirements
- CoreEvery service’s metrics are collected, stored and can be graphed over time.
- CoreEngineers query any metric over a time range, added up or broken down by its labels.
- CoreAlert rules watch the metrics and page the right on-call engineer.
- CoreEach team owns its metrics, dashboards and alert rules.
Non-functional requirements
- LatencyReporting a metric never slows the service reporting it.
- LatencyA new sample can be queried and alerted on within about 30 seconds.
- AvailabilityAlerting keeps working when a region fails.
- DurabilityA single machine failure loses no samples.
- AvailabilityOne team’s metrics can’t make monitoring worse for other teams.
- CostEach team can see what its metrics cost.
Scale
- ScaleAbout 100M active series, each sampled every 15 seconds.
- ScaleAbout 50,000 alert rules, evaluated every 30 seconds.
- ScaleAbout 1,000 dashboard queries a second during a big incident.
- ScaleThe last 15 days in full detail; 13 months for longer graphs.
Nice to mention (below the line)
- Below the lineLogs and traces, beyond linking to them.
- Below the lineBuilding the paging and on-call scheduling tool.
- Below the lineAnomaly detection.
Out of scope
- Out of scopeLogs and traces.
- Out of scopeThe paging and on-call scheduling tool.
- Out of scopeAnomaly detection and service discovery.
The variant
One fully specified brief, with the level it suits best. You can practise it at any level.
- Monitoring for a large companySuggested for Senior
Collect, store and alert on the metrics of 2,000 services run by 500 teams, without slowing them down or going quiet when they fail.
8 steps · 5 must-pass · about 48 min at Senior
What each level is graded on
Monitoring for a large company, step by step. A result at a level needs its must-pass steps answered, and only the steps you answer count. Going up a level adds steps or holds them to a stricter bar.
| Step | Entry level | Mid-level | Your levelSenior | Staff |
|---|---|---|---|---|
| The numbers that shape the design (optional) | Graded | Graded | Graded | Graded |
| What a metric is, and how it’s stored | Graded | Graded | Graded | Graded |
| The API | Graded | Graded | Graded | Graded |
| From a counter to a dashboard, and to a page | Must-pass | Must-passcan block | Must-passcan block | Must-passcan block |
| Seven million samples a second | Must-pass | Must-pass | Must-pass | Must-pass |
| The label that took down the store | Must-passcan block | Must-passcan block | Must-passcan block | Must-passcan block |
| The alert that cried wolf | — | Must-pass | Must-pass | Must-pass |
| The night the monitoring went dark | — | — | Must-pass | Must-pass |
| Every team, every region | — | — | — | Must-pass |
| At this level | 6 steps 3 must-pass about 34 min | 7 steps 4 must-pass about 41 min | 8 steps 5 must-pass about 48 min | 9 steps 6 must-pass about 56 min |
Entry level6 steps · 3 must-pass · about 34 min
- The numbers that shape the design (optional)Graded
- What a metric is, and how it’s storedGraded
- The APIGraded
- From a counter to a dashboard, and to a pageMust-pass
- Seven million samples a secondMust-pass
- The label that took down the storeMust-passcan block
Mid-level7 steps · 4 must-pass · about 41 min
- The numbers that shape the design (optional)Graded
- What a metric is, and how it’s storedGraded
- The APIGraded
- From a counter to a dashboard, and to a pageMust-passcan block
- Seven million samples a secondMust-pass
- The label that took down the storeMust-passcan block
- The alert that cried wolfMust-pass
SeniorYour level8 steps · 5 must-pass · about 48 min
- The numbers that shape the design (optional)Graded
- What a metric is, and how it’s storedGraded
- The APIGraded
- From a counter to a dashboard, and to a pageMust-passcan block
- Seven million samples a secondMust-pass
- The label that took down the storeMust-passcan block
- The alert that cried wolfMust-pass
- The night the monitoring went darkMust-pass
Staff9 steps · 6 must-pass · about 56 min
- The numbers that shape the design (optional)Graded
- What a metric is, and how it’s storedGraded
- The APIGraded
- From a counter to a dashboard, and to a pageMust-passcan block
- Seven million samples a secondMust-pass
- The label that took down the storeMust-passcan block
- The alert that cried wolfMust-pass
- The night the monitoring went darkMust-pass
- Every team, every regionMust-pass
- Must-pass
- a result at that level needs this step answered.
- Can block
- a miss there is a Blocking issue at that level.
Going up a level
- Mid-level
- A miss on “From a counter to a dashboard, and to a page” can now block the result.Adds “The alert that cried wolf”, must-pass.
- Senior
- Adds “The night the monitoring went dark”, must-pass.
- Staff
- “The alert that cried wolf” is graded more strictly.Adds “Every team, every region”, must-pass.
Skills this problem practises
Your coach tracks readiness per skill across every problem you practise. Each links to the lesson that teaches it.
- Capacity estimation
- API design
- Data modelling
- Choosing storage
- Sharding and partitioning
- High-level architecture
- Queues and async messaging
- Traffic surges and backpressure
- Failure modes and recovery
- Observability and operations
Worked solution
Try it first: practise a variant, then read the walkthrough and compare it with your own design. Reading it first makes the practice much less useful.
Learn: Design a metrics and monitoring systemYour history
Sign in to see your sessions on this problem.