Design an Event Metrics Aggregation System with IP Geo Enrichment

Design a metrics aggregation system. Clients produce a large number of events, such as user clicks, across many event types. The system must take the events in, and a dashboard queries aggregate counts over them. The reported requirements: 1. Collect event data: 1M events and 10K event types. 2. Serve dashboard queries such as the number of click events within a day. 3. Add context to each event, for example use the client's IP address to identify the location of the event. 4. Serve more complex queries, such as the number of click events in the US region over the last 3 months. All four requirements are in scope. In the reported round, time ran out before the fourth was discussed, so plan your time to reach it: it shapes the storage design. ```hint Start from the queries For each dashboard query, decide what pre-computed data it would read and at what time granularity, before choosing where events are stored. ``` ```hint Enrich once Decide where the IP-to-location lookup happens so that no later query needs the raw IP, and what happens to past events when the lookup data changes. ``` ### Constraints and Clarifications - The event volume was given as "1M events" without a time unit; resolve it before estimating. - There are about 10,000 distinct event types. - Location comes from the IP address the event was sent from. ### Clarifying Questions - Is the volume 1 million events per second, per minute or per day, and how high is the peak relative to the average? - How fresh must dashboard counts be, and must they be exact? - Besides event type and location, which dimensions can a query filter or group by? Are unique users needed, or only event counts? - Which time zone defines "a day", and how late can events arrive? - How long must raw events and aggregates be kept? - May the raw IP address be stored, or only the derived location? ### What a Strong Answer Covers - Requirements and a capacity estimate that resolves the unit of the event volume - Durable, partitioned ingestion with deduplication and back-pressure - IP enrichment at ingestion, with versioned lookup data and privacy for raw IP addresses - Time-bucketed rollups by event type and location for dashboard reads, plus raw storage for ad hoc queries and backfills - Late, duplicate and out-of-order events, a dominant event type, and retention by granularity - A query path for multi-month, multi-dimension queries and its latency ### Follow-up Questions - The dashboard now needs unique users per event type per day instead of event counts. What changes? - One event type accounts for most of the traffic. Where does it create a hot spot, and how do you spread the load? - The IP-to-location data is updated and some address ranges move to another country. Do you restate past counts? - A new dimension, such as device type, is wanted on three months of history. How do you provide it?