Design a Sliding-Hour Top-100 User Event Pipeline

# Design a Sliding-Hour Top-100 User Event Pipeline An event stream contains records `(user_id, event_type, timestamp)`. Design a system that continuously returns the 100 users with the most events in the trailing hour. Present both a high-level architecture and low-level pseudocode for the windowed counting logic. Compare batch processing with streaming, and compare a record-at-a-time engine such as Flink with a micro-batch engine such as Spark Structured Streaming for this workload. Address scalability, event time, late and duplicate records, state size, exactness, tie ordering, recovery, and result serving. ### Clarifying Questions to Ask - Is “trailing hour” continuously evaluated or refreshed at fixed intervals? - How late can events arrive, and may corrected results replace earlier results? - Must counts and the top 100 be exact? - What deterministic ordering resolves equal counts? ### What a Strong Answer Covers - Durable ingestion and partitioning with a stable event identity. - Event-time sliding-window semantics, watermarks, and explicit lateness policy. - Bucketed per-user counts with expiration and a scalable top-100 aggregation. - A grounded batch-versus-streaming and Flink-versus-Spark comparison. - Checkpointed state, replay-safe updates, backpressure, skew, and serving snapshots. - Low-level pseudocode whose state transitions match the stated window semantics. ### Follow-up Questions - How would you support an exact answer refreshed every second without scanning every user? - What changes if one user produces a large fraction of all events? - How would you verify a recovered streaming result against a batch recomputation?