## Design Global Metrics Monitoring and Real-Time Alerting Design a large-scale monitoring and alerting system for a global microservice architecture. The system must track service health, visualize performance through dashboards, and send real-time alerts when anomalies occur. ### Part 1 — Define Health and the Metric Contract Specify metric types, labels, timestamps, service and region identity, service-level indicators, and the meaning of real-time alerting. Identify required ingestion rate, active-series count, retention, dashboard latency, and alert delay. #### What This Part Should Cover - Counters, gauges, distributions, and canonical series identity. - Tenant, service, instance, and region dimensions without unbounded labels. - Explicit freshness and late-data semantics. - Health objectives based on measurable signals rather than host uptime alone. ```hint Start from decisions, not storage Define which service-health questions dashboards and alerts must answer before selecting a time-series layout. ``` ### Part 2 — Ingest, Store, and Query Globally Design the producer, gateway, durable-ingestion, time-series storage, indexing, dashboard-query, retention, and downsampling paths. Explain regional failure behavior and global query consistency. #### What This Part Should Cover - Local regional ingestion with a clear durability acknowledgement. - Partitioning by tenant or series and time without permanent hot shards. - Label-to-series lookup separated from time-value storage. - Bounded fan-out, partial-result policy, and regional freshness metadata. ```hint Keep collection close to the source Regional ingestion can survive a cross-region link failure, but global views must reveal which regions have not replicated recently. ``` ### Part 3 — Evaluate Alerts and Operate the Platform Design stateful rule evaluation, anomaly detection integration, notification delivery, deduplication, silencing, and self-monitoring. Handle overload, missing shards, and the monitoring system failing while services are unhealthy. #### What This Part Should Cover - Rule ownership, evaluation windows, and pending or firing state. - Retry-safe notification fingerprints and escalation policy. - Unknown-data behavior rather than treating missing data as healthy. - Independent health signals for ingest, storage, queries, rules, and notifications. ```hint Alert on freshness as well as values A normal-looking value is not trustworthy when the newest sample is older than the evaluation window. ``` ### What a Strong Answer Covers - Quantified health, dashboard, and alert requirements. - Region-aware durable ingestion and scalable time-series storage. - Cardinality control, retention tiers, and honest partial-query semantics. - Stateful, deduplicated alerting plus an independent operational health path. ### Follow-up Questions 1. How would you diagnose one region whose dashboards are current but alerts are delayed? 2. What should happen when a deployment creates millions of new label combinations? 3. How would you evaluate an anomaly model before allowing it to page operators? 4. Which data must cross regions, and which can remain regional?