## Scenario Design a monitoring platform for count metrics. Producers emit events such as `request_completed` with a timestamp and a bounded set of labels. Users can query counts and rates over recent time ranges, group by approved labels, and create threshold alerts. Cover ingestion, aggregation, storage, querying, alert evaluation, late and duplicate events, retention, cardinality protection, and observability. No traffic numbers are supplied; identify which choices require them and present a coherent baseline. ### Constraints & Assumptions - The core metric type is a nonnegative count; gauges, histograms, and traces are extensions. - Producers may retry, events may arrive late, and clocks may be skewed. - Label values can explode cardinality unless the service enforces limits. - Losing acknowledged events silently is unacceptable, but a short query freshness delay is allowed as an explicit assumption. ### Clarifying Questions to Ask - What event rate, active-series count, retention, and query concurrency are expected? - Are counts exact, and what lateness window should be supported? - Which labels are permitted and how are tenants isolated? - How quickly must an alert react, and how are missing data and delayed data interpreted? ```hint Separate durable ingestion from serving aggregates Accept to a partitioned log, aggregate by event-time buckets, and write query-optimized rollups. The raw log gives replay; the rollups give predictable reads. ``` ### What a Strong Answer Covers - A stable series identity from metric name plus normalized labels. - Durable partitioned ingestion, idempotency choices, event-time windows, watermarks, and correction of late buckets. - Time-bucket rollups, retention tiers, query planning, caching, and tenant quotas. - Alert state machines with evaluation intervals, consecutive-window rules, and notification deduplication. - Cardinality limits, backpressure, load shedding policy, and end-to-end lag and reconciliation metrics. ### Follow-up Questions 1. How would you prevent one unbounded label from exhausting the service? 2. What does an alert evaluator do when a bucket is incomplete or missing? 3. How would you rebuild aggregates after discovering a faulty counting rule?