Design a Five-Minute Top-K Error Log Service

# Design a Five-Minute Top-K Error Log Service Design and outline the core implementation of a log collection system that ingests logs from many microservices and returns the `K` services with the most error logs in the trailing five minutes. The design should handle a very large log volume while meeting explicit scalability, availability, and fault-tolerance goals. Define error classification, window refresh semantics, exactness, and tie ordering. Then cover the ingestion path, aggregation state, top-K computation, result serving, retries, overload behavior, and low-level state transitions. ### Clarifying Questions to Ask - How are error logs identified, and may classification rules change? - How often must the top-K result refresh, and how late can logs arrive? - Is an exact result required, and how should equal counts be ordered? - What availability and recovery objectives apply to ingestion and queries? ### What a Strong Answer Covers - A durable, partitioned ingestion path with stable event and service identifiers. - Event-time buckets that expire from a five-minute rolling count. - An exact local-candidate and global-merge strategy, or a clearly bounded approximation. - Idempotent processing, checkpoint recovery, late data, skew, and backpressure. - Separation between log durability, aggregation, and low-latency result serving. - Core pseudocode plus metrics for loss, lag, correctness, and result age. ### Follow-up Questions - How would a sudden error storm from one service affect partitions and downstream storage? - How would you change error-classification rules without silently mixing definitions in one window? - What evidence would prove that a replay did not double-count logs?