## Design a Scalable Metrics Monitoring System Design a service that accepts application and infrastructure metrics, stores them over time, supports dashboards and ad hoc queries, and evaluates alert rules. Address scale, freshness, cost, and operational reliability. ### Constraints & Assumptions - Producers emit counters, gauges, and histogram-like distributions with timestamps and label sets. - Label cardinality can dominate storage and query cost, so the system must make it visible and controllable. - Recent data needs lower query latency and finer resolution than old data. - Delayed, duplicated, and out-of-order samples are possible. - Required ingestion rate, retention, query latency, and alert delay must be clarified rather than guessed. ### Clarifying Questions to Ask - How many samples per second, active series, tenants, and regions are expected? - Which query operations are required: range scans, aggregation, percentiles, joins, or arbitrary formulas? - How fresh must dashboards and alerts be, and how much late data is accepted? - What retention and downsampling policy applies to each metric class? - Which tenant isolation, access control, and availability targets are mandatory? ### Part 1 — Define the Metric and Write Contract Define series identity, sample format, batching, timestamps, duplicate handling, and producer feedback. Explain how the design prevents unbounded label cardinality. #### What This Part Should Cover - Metric name, type, labels, timestamp, and value representation. - Stable series IDs and tenant scoping. - Idempotent or deterministic duplicate handling. - Limits, rejection signals, and visibility for high-cardinality labels. ```hint Separate a series from a sample The label set identifies the time series; timestamps and values are repeated observations within that identity. ``` ### Part 2 — Ingest, Store, and Query Design the path from producer batches to durable time-series storage and query results. Include partitioning, indexing, replication, compaction, and downsampling. #### What This Part Should Cover - Load distribution by tenant and series without creating permanent hot shards. - A write-ahead or replicated durability boundary. - Time-oriented data blocks plus an index from labels to series. - Query fan-out, partial-result policy, caching, and retention tiers. ```hint Use different layouts for discovery and values Finding which series match a label expression and scanning samples for those series have different access patterns. ``` ### Part 3 — Alert and Operate at Scale Explain how alert rules are scheduled and evaluated, how notifications are deduplicated, and how the monitoring system monitors itself during overload or partial failure. #### What This Part Should Cover - Rule ownership, evaluation windows, and state for pending or firing alerts. - Retry-safe notification delivery and silencing. - Backpressure and graceful degradation during ingestion or query spikes. - Metrics for dropped samples, lag, cardinality, query fan-out, and alert delay. ```hint Measure each freshness boundary Producer-to-ingest lag, durable-write lag, query visibility, rule evaluation, and notification delivery can fail independently. ``` ### What a Strong Answer Covers - Quantified requirements and a precise series and sample contract. - Horizontally partitioned ingestion, durable time-series storage, and an appropriate label index. - Explicit late-data, duplicate, retention, and high-cardinality policies. - Stateful, retry-safe alerting plus self-observation and overload behavior. ### Follow-up Questions 1. How would you handle one tenant creating millions of unique label combinations per minute? 2. What consistency should a dashboard expect immediately after a successful write? 3. How would you compute long-range percentiles without keeping every raw sample forever? 4. What should alerts do when one storage shard is unavailable?