Design a Distributed Log Pipeline That Counts Keywords Across a Fleet

After two short coding problems, the interviewer extended the discussion into a design question: many machines produce logs, and you need a distributed system that collects those logs and counts the keywords that appear in them. The keyword set, the log volume and how fresh the counts must be were not specified. Ask, or state your assumptions before designing. ### Clarifying Questions - Is there a fixed list of keywords to track (for example, terms such as `ERROR` or `timeout` configured by operators), or should every word be counted so that the most frequent terms can be reported? - How fresh must the counts be: seconds behind the logs, or is an hourly or daily batch enough? - How should counts be broken down: by time bucket, service, host, log level? - How many machines, how many log lines per second, and how long must counts and raw logs be kept? - What counts as a match: whole word or substring, case-sensitive or not? Are the logs free text or structured? - Must counts be exact, or is a small, bounded error acceptable? ### Part 1 — Getting logs off the machines Describe how log lines travel from the machines that write them to the place where they are counted, and what happens when a machine, the network, or a downstream component fails. ```hint Downstream will sometimes be down Decide where log lines wait when the next stage is slow or unavailable for several minutes, and how a machine knows that a line has been safely handed off. ``` #### What This Part Should Cover - The collection agent on each machine: batching, retries and local buffering - A durable, partitioned transport between collection and processing, and its retention - Delivery guarantees (at-most-once versus at-least-once) and the duplicates they imply - Backpressure, and what is lost, if anything, when buffers fill ### Part 2 — Counting at scale No single machine can read every log line. Design the computation that turns the stream of lines into keyword counts. ```hint Split the work, then combine Think about which part of the counting each worker can do on its own share of the lines, and what has to be merged across workers afterwards. ``` ```hint One keyword can dominate During an incident, one term may appear in a large fraction of all lines. Consider what happens to whichever component is responsible for that term. ``` #### What This Part Should Cover - Matching keywords in each line efficiently, and how changes to the keyword list take effect - Partitioning and partial aggregation, so the traffic per keyword stays bounded - Time buckets based on when a line was written, and handling of late or out-of-order lines - Hot keywords and skew ### Part 3 — Storing, serving and correcting the counts Where do the counts live, how are they queried, and how do you keep them correct when workers crash and lines are delivered twice? ```hint Replays are normal Assume that any batch of lines may be processed twice. Decide how a count update can be applied more than once without inflating the total. ``` #### What This Part Should Cover - Storage layout for counts, and rollups by time granularity - The query API, and how long-range queries stay fast - Idempotent or exactly-once updates tied to processing checkpoints - Recomputing history when the keyword list changes or a bug is found ### What a Strong Answer Covers - Pins down keyword semantics, freshness and volume before choosing an architecture - Back-of-the-envelope estimates that drive partition counts and storage - An end-to-end data flow from agent to query, with explicit delivery guarantees - Correctness under crashes, duplicate deliveries and late data - Observability: lag, lost lines, and checks that the counts match the raw logs - Trade-offs between streaming and batch, and between exact and approximate counting ### Follow-up Questions - Operators add a new keyword and want its counts for the past 30 days. How do you produce them? - Instead of a fixed list, report the 100 most frequent words per hour with bounded memory. What changes, and what accuracy do you give up? - How would you verify, end to end, that the counts are right? - A whole data center loses connectivity for an hour and then reconnects. What happens to its logs and to the counts for that hour?