Design a metrics and monitoring system

Every service reports numbers about itself, engineers graph them, and rules page someone when something is wrong. The difficulty is volume and trust: about 100M series report every 15 seconds, so the write path dwarfs everything else, and monitoring is only useful if it keeps working, and keeps quiet, at exactly the moments the rest of the company is failing.

The practice brief states the requirements up front; confirm them in the first few minutes. The core is a write path that doesn’t slow the services, storage that fits a year of graphs into a budget, and alerts people can trust.

Work out only the numbers the design depends on, with rough figures. The one that shapes everything is the series count.

A metric is a name with a type, a unit and an owner. A series is one metric plus one set of label values, such as service, endpoint and status; every distinct set is its own series with its own id. A sample is a time and a value, appended to its series.

A label index maps each label value to the series that carry it, so a query for checkout errors intersects two lists instead of scanning everything. Counters only go up and are read as rates, so a restart’s reset to zero is handled when the rate is taken.

Each caller gets its own contract. Long-running services show their counters for a collector to read; short batch jobs, which exit before anyone would collect them, push a batch instead (pushing docs (opens in a new tab)). Dashboards ask for a range at a step that matches the pixels they can draw. Teams deploy alert rules as reviewed code.

Build the design one path at a time: samples in, graphs out, pages out. The deep dives then harden the parts an interviewer pushes on.