Design a Metrics and Alerting System, Focusing on How Metrics Are Stored

Design a system that collects metrics from services and hosts and raises alerts when those metrics meet conditions that operators define. The interviewer focused on how the metrics are stored, so expect most of the discussion to be about the storage layer: the data model, how samples are written and compressed, how long they are kept and at what resolution, and how dashboards and alert rules read them back. ```hint What a series is Decide what identifies one time series and what one stored sample contains, then ask which of those parts repeats with every sample and which does not. ``` ```hint Writes and reads have different shapes Samples arrive for millions of series at once, all near the current time, while a dashboard or an alert reads a few metrics over a time range. Choose a layout that serves both. ``` ### Constraints and Clarifications - Not given: the number of sources and series, the collection interval, retention, query patterns, and how fast an alert must fire. Ask, or state your assumptions. ### Clarifying Questions - What kinds of metrics are collected: counters, gauges, latency histograms? Do they carry labels such as service, host and region, and how many distinct values can a label have? - How many hosts and services, how many series in total, and how often is each sampled? Do sources push, or does the system pull? - How long must data be kept, and at what resolution for recent versus old data? - What do queries look like: dashboards over ranges, aggregation across labels, percentiles? How fast must they return? - How soon after a condition becomes true must the alert fire, and where do notifications go? - Is losing a few samples during a failure acceptable? Must alerting keep working when the storage is degraded? ### What a Strong Answer Covers - Settles metric types, cardinality, resolution, retention and alert latency before designing - A series model (metric name plus labels) kept separate from the stream of samples, with an index from labels to series - A storage engine built for append-heavy, time-ordered writes: buffering, durability, time-partitioned immutable data and compression suited to timestamps and values - Downsampling and retention tiers that keep statistics which can still be aggregated correctly - Sharding, replication and a query path for range queries and aggregation across series - Alert rule evaluation, alert state, deduplication, grouping and delivery, and how the alerting pipeline itself is monitored - Cardinality control, missing data and other failure modes ### Follow-up Questions - A team adds a user-id label to a request-count metric. What happens to your storage, and how do you protect the system? - How do you compute the 99th-percentile latency across 500 hosts over the last week from stored data? - An ingestion node is down for ten minutes. What do dashboards and alerts show, and how do you avoid false or missed alerts? - How would you alert when a host stops reporting entirely?