# Design a Low-Latency Metrics and Alerting Platform Design a multi-tenant platform that collects host and application metrics, stores time-series data, serves dashboards and queries, evaluates service-level objectives, and sends alerts. Important alerts should detect failures within two seconds, including the failure of an individual machine worker. ### Constraints & Assumptions - Metrics contain a name, timestamp, numeric value, and bounded label set. - Hosts emit heartbeats in addition to application metrics. - Data can arrive late, out of order, duplicated, or during regional failures. - Recent high-resolution data has a shorter retention period than downsampled historical data. - Alert delivery providers and parts of the monitoring platform can fail. ### Clarifying Questions to Ask - What ingestion rate, active-series cardinality, retention, query volume, and tenant count are expected? - Does the two-second target measure detection, evaluation, or successful notification delivery? - Which metric types and query operations must be supported? - How much late data may revise dashboards or alert evaluations? - What false-positive and false-negative trade-off is acceptable for machine failure detection? ### Hints - Separate ingestion, durable storage, query, rule evaluation, and notification concerns. - Heartbeat absence needs an expected cadence and failure threshold, not just a missing row. - Treat alert state as durable data so evaluator retries do not duplicate notifications. ### What a Strong Answer Covers - Instrumentation, agents, collectors, authentication, batching, validation, deduplication, and backpressure. - Partitioning and indexing of time-series data, cardinality controls, retention, compaction, and downsampling. - Query fan-out, recent-data caches, aggregation, limits, and tenant isolation. - Versioned rule and SLO configuration, incremental evaluation, alert state machines, silences, grouping, and deduplicated delivery. - Heartbeats, lease-style liveness, jitter, maintenance awareness, correlated failures, and false-positive control. - High availability, regional failure behavior, observability of the observer, cost, and a defensible two-second latency budget. ### Follow-up Questions 1. How do you detect a failed worker when the network between collectors and storage is partitioned? 2. How would you prevent one tenant's high-cardinality labels from exhausting the platform? 3. What happens when a late sample changes a previously evaluated alert window? 4. How do you test alert delivery when the metrics platform itself is degraded?