## Problem Design a **User Behavior Tracking System** that collects and analyzes user events from **mobile and web applications** across multiple products. The system must capture events such as `page_view`, `click`, and `purchase`, expose **client SDKs** (web + mobile) and **server-side API endpoints** for collection, and serve two very different analytics workloads from the same instrumented data: 1. **Dashboard queries (interactive)** — fast response times for common metrics and slices (DAU/MAU, event counts, funnels). 2. **Deep analytics / warehousing queries** — complex, ad-hoc queries over large historical data, which may run for minutes. The system must also support **data enrichment**, including reverse geolocation from IP/GPS and compliance/policy tagging (e.g., PII flags, data-residency constraints). Your design should cover the high-level architecture and components, the event schema (and how it evolves), the SDK→backend ingestion pipeline and its reliability guarantees, the storage choices for dashboard vs. warehouse workloads, the enrichment design (real-time vs. batch), and the key tradeoffs, bottlenecks, and operational concerns (monitoring, data quality, privacy). ```hint Where to start Before drawing any boxes, put rough numbers on the workload: a peak events/sec, an average event size, and a retention window. Then compare the two things the system has to do — absorb the write firehose vs. answer dashboard reads — and ask which one dominates the design. What does that comparison suggest about whether events should land *directly* on whatever database serves your queries? ``` ```hint Buffering the firehose You've sized a large, bursty write rate against a comparatively modest read rate. What sits between the stateless collectors and the rest of the system to absorb those bursts, let multiple downstream consumers read the same events independently, and let you re-process history after a bug? Name the category of component and the properties you need from it. ``` ```hint Two reads, two shapes The two query paths want *opposite* things from storage — fast point slices on recent data vs. cheap massive scans over months. Rather than compromise on one store, consider how many serving stores you actually need and what each is optimized for. How can you populate them without instrumenting or ingesting events twice? ``` ```hint Schema & delivery pitfalls Two traps to reason about. (1) Schema churn: how do you let product teams add fields without breaking existing readers — what structural split in the event, plus what enforcement mechanism, gets you that? (2) Duplicates and disorder: weigh chasing exactly-once against a simpler delivery guarantee paired with dedup, and think about *which* timestamp you window on when client clocks drift and devices replay offline data. ``` ```hint Enrichment placement Some enrichment is lightweight and hot-path (e.g., GeoIP, user-agent parsing); some is heavy or changes often. Where should each kind run, and how do you avoid a network round-trip per event for the reference data the hot path needs? Separately: what property of your raw events makes a later logic change safe to re-apply over history? ``` ### Constraints & Assumptions State your assumptions explicitly; the following are reasonable defaults to design against (tune to the interviewer's numbers): - **Scale**: on the order of tens of thousands of events/sec average with several-fold peaks (e.g., ~50K EPS average, ~250K EPS peak); average serialized event ~1 KB. - **Retention**: short hot window for interactive dashboards (e.g., days–weeks), long cold retention in the lake (e.g., ~13 months). - **Freshness SLA**: dashboards may tolerate ~1–5 minutes of lag; not strictly real-time. - **Latency target**: dashboard P95 in the low seconds (ideally sub-second for canonical metrics). - **Delivery**: at-least-once is acceptable provided downstream dedup makes counts correct. - **Cross-cutting**: reliability (no silent data loss), horizontal scalability, schema evolution, access control, and privacy/compliance are required, not optional. ### Clarifying Questions to Ask - What is the **peak events/sec**, average event size, and number of distinct products/apps feeding the system? - What is the **freshness SLA** for dashboards (seconds vs. a few minutes) and the **P95 latency** target? - How long must **raw history** be retained, and what are the hot vs. cold tiers? - Do we need to **stitch identity** (anonymous → logged-in, and across devices), and what defines a "unique user" for DAU/MAU? - What **compliance** obligations apply (GDPR/CCPA deletion, data residency by region)? - Is **exactly-once** required, or is at-least-once with idempotent dedup acceptable? ### What a Strong Answer Covers - **Capacity estimate**: write throughput, daily/retained volume, and the read-vs-write asymmetry that motivates the architecture. - **Architecture**: a clear data flow (SDK → edge/collector → durable log → stream processor → serving stores → query APIs) with each component's role justified. - **Decoupling**: explicit recognition of the write/read asymmetry and a buffered, fan-out design rather than direct-to-DB writes. - **Data model**: a stable event core plus a flexible `properties` bag; separate client vs. server timestamps; stable IDs for dedup and identity. - **Schema evolution**: a registry with compatibility rules, versioned events, and a dead-letter/quarantine path (no silent drops). - **Ingestion reliability**: SDK batching/offline buffering/retry; stateless autoscaling collectors; partitioning strategy and hot-key handling; clearly stated delivery semantics. - **Dual storage**: an OLAP store (rollups, cardinality control, time partitioning) for dashboards and a lakehouse for async analytics, with sync vs. async query APIs. - **Enrichment**: a real-time-vs-batch tradeoff with a defensible hybrid, plus reproducibility/backfill via versioning and immutable raw. - **Operations & privacy**: monitoring (consumer lag, freshness, drop/dup rates, cardinality), data-quality checks, PII handling, deletion, residency, and access control. ### Follow-up Questions - A single **"whale" tenant** produces a disproportionate share of events and creates a hot partition. How do you detect and mitigate this without losing per-actor ordering? - A user is active **anonymously, then logs in**. How do you attribute the earlier anonymous events to that user, and what is the latency/cost tradeoff of doing so? - You change the **compliance-tagging logic**. How do you re-derive enriched fields over historical data without corrupting existing dashboards, and how do consumers know which version they're reading? - The dashboard for one product **slows from sub-second to many seconds overnight**. Walk through how you'd diagnose it (e.g., cardinality explosion, consumer lag, segment skew).