### Prompt The interview report mentioned detailed questions about the candidate's batch- and stream-processing experience, but it did not preserve an exact scenario. The following is a **self-contained practice reconstruction** of that topic. You are designing a data platform for an online marketplace. Order lifecycle events arrive continuously and may be duplicated, delayed, or delivered out of order. Product teams need operational metrics within five minutes, while the finance team needs a reproducible daily table by the following morning. Historical corrections and event-schema changes are expected. Explain whether you would use batch processing, stream processing, or a hybrid design. Describe the end-to-end architecture and how it would produce both low-latency metrics and an auditable daily result. ### Constraints & Assumptions - Events have a stable event identifier, an event-time timestamp, and an order identifier. - The source can replay retained events, but consumers should not assume exactly-once network delivery. - The operational view may be eventually consistent; the finance table must be reproducible for a declared input version. - You may choose technologies by capability, but the answer should not depend on a particular vendor. - Treat privacy, access control, and deletion requirements as part of the design. ### Clarifying Questions to Ask - What volume, peak throughput, retention period, and acceptable recovery time should the system support? - What does “within five minutes” measure: event time, ingestion time, or dashboard refresh time? - How late can events arrive, and how should finalized periods handle later corrections? - Which metrics require joins with mutable reference data? - Must downstream consumers observe updates, retractions, or only append-only versions? ### Part 1: Choose the Processing Model Compare batch, streaming, and hybrid approaches against latency, correctness, operational complexity, replay, and cost. Make a recommendation and state what evidence could change it. #### Hints Start from the two service-level needs rather than treating “streaming” as the default. Separate the speed of an answer from the authority of an answer. #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 2: Design the Data Flow Describe ingestion, durable storage, validation, deduplication, event-time handling, stateful computation, batch finalization, metadata, and serving. Explain how data contracts and schema evolution work. #### Hints Keep immutable raw input so that corrected logic can be replayed. Define idempotency at every write boundary. #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 3: Operate and Validate the Pipeline Explain how you would deploy changes, detect correctness failures, recover from them, and verify that the low-latency and finalized views agree within their documented semantics. #### Hints Monitor data behavior as well as infrastructure behavior. A pipeline can be healthy at the service level while producing wrong numbers. #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### What a Strong Answer Covers ```premium-lock What a Strong Answer Covers ``` ### Follow-up Questions 1. When would a batch-only design be the better engineering decision? 2. How would you join the stream to a customer dimension that changes over time? 3. What happens when an event arrives after the watermark and after finance has closed the day? 4. How would you migrate a stateful aggregation without double-counting? 5. Which guarantees are provided by the transport, the processor, and the sink respectively?