Design an Internal Metrics Ingestion Service

# Design an Internal Metrics Ingestion Service Design an internal service that collects metrics through client-side agents and supports reliable ingestion plus later queries. Focus on agent behavior, sharding, availability, and decoupling the write path from the query path. ### Constraints & Assumptions - Agents may lose connectivity and retry, creating duplicates. - Metric cardinality can grow unexpectedly. - Recent data and historical aggregates have different latency and storage needs. ### Clarifying Questions to Ask - What metric types, labels, ingestion volume, retention, and query latency are required? - Is at-least-once delivery acceptable, and how much data loss can be tolerated? - Which tenant or service isolation rules apply? ```hint Protect the write path from query load An append-oriented ingestion pipeline and a read-optimized serving layer can evolve independently when a durable boundary separates them. ``` ### What a Strong Answer Covers - Agent buffering, batching, backpressure, retry, and identity. - Durable ingestion, partitioning, replication, and overload controls. - Aggregation, compaction, retention, and query indexes. - Cardinality limits, observability, multi-tenancy, and failure recovery. ### Follow-up Questions - How would you handle a client clock that is far from server time? - What happens when one label creates millions of time series?