Design a pipeline that ingests a live data stream and powers real-time charts

A later round in this loop was mostly behavioral but included one high-level design question about a live data stream and its visualization. Design a system that ingests a continuous stream of data and lets users watch it on charts that update as new data arrives. The data domain was not specified, so assume a general one: many producers continuously send timestamped events (for example, application metrics or events derived from logs), and users open dashboards whose charts show the last few minutes updating live as well as longer history. Presentation carried weight in this round. You may have to explain the whole design out loud without drawing anything, although some interviewers ask you to sketch it in an online editor, so practice both. ```hint Decide where aggregation happens A browser cannot draw millions of raw points per second; decide which layer turns raw events into something a chart can display. ``` ```hint Narrate in layers Without a diagram, give the listener a map first (the stages in order), then go deep on one stage at a time. ``` ### Constraints and Clarifications - Event volume, freshness target and retention are not given; state your own estimates and targets. - Assume many users can watch the same dashboard at the same time. - Expect to present the design clearly with or without a diagram. ### Clarifying Questions - Who produces the stream, and roughly how many events per second arrive at peak? - How fresh must the live chart be, and may it show pre-aggregated or approximate values rather than every raw event? - How far back do users need to look, and at what resolution for older data? - How many people watch the same dashboard at once, and must they all see identical values? - How should late or out-of-order events show up on a chart that has already been drawn? ### What a Strong Answer Covers - A clear end-to-end path: ingestion, durable buffering, stream processing, storage, query, and push to the browser - Windowed aggregation and downsampling so clients receive data at a rate a chart can use - Handling of late, duplicate and out-of-order events, with an explicit policy for revising points already drawn - Storage for recent versus historical data, with a retention and rollup plan - Backpressure and failure handling across the pipeline, plus monitoring of end-to-end lag - A verbal walkthrough that is structured and easy to follow without a diagram ### Follow-up Questions - One producer suddenly sends 50 times its normal volume. How does the pipeline protect everyone else's dashboards? - A user needs an exact count for the last hour rather than an approximation. What changes? - How would you add alerting on the same stream without building a second pipeline? - Explain the design again in two minutes for a non-technical stakeholder. What do you say?