Design a Metrics Collection System

# Design a Metrics Collection System Design a general metrics collection system. State the metric model and collection contract, then cover ingestion, aggregation, storage, query, retention, reliability, and operational safeguards. ### Constraints & Assumptions - Metrics may arrive at high volume and out of order. - Labels can create unbounded cardinality if not governed. - The system should distinguish telemetry loss from a real drop in the measured service. ### Clarifying Questions to Ask - Are metrics counters, gauges, histograms, events, or a defined subset? - What query windows, freshness, and retention are required? - How much loss or duplication is acceptable? ```hint Control cardinality early Define allowed labels and enforcement before scaling storage or query tiers. ``` ### What a Strong Answer Covers - Client and gateway protocol, batching, identity, and timestamps. - Partitioning, aggregation, durable buffering, storage tiers, and retention. - Query and dashboard path with stable semantics. - Backpressure, deduplication, cardinality limits, monitoring, and disaster recovery. ### Follow-up Questions 1. How would you support accurate percentiles without storing every sample? 2. How would multi-region ingestion preserve availability and understandable counts?