### Prompt Choose one production data pipeline that you personally designed or substantially changed. Walk through its architecture, the requirements that drove it, the technology choices and alternatives you considered, one difficult technical challenge, and what you learned from operating it. The goal is not to list tools. Demonstrate ownership by connecting each major choice to a constraint, evidence, and outcome. If confidentiality limits detail, use sanitized names and scale ranges rather than inventing facts. ### Constraints & Assumptions - Focus on one coherent pipeline rather than combining unrelated projects. - Distinguish decisions you owned from decisions inherited or made by the team. - Quantify real requirements and outcomes where you can defend them; otherwise state the qualitative constraint. - Cover correctness, reliability, cost, security, and maintainability, not just the happy-path data flow. - Discuss at least one rejected alternative and one decision you would revisit today. ### Clarifying Questions to Ask - Should the discussion emphasize architecture, incident response, or team leadership? - How much implementation detail is useful for the role and available time? - May the architecture be drawn, and should current or original state be shown? - Are sanitized internal details acceptable? - Which dimension matters most to the interviewer: latency, scale, correctness, cost, or developer velocity? ### Part 1: Establish Context and Requirements Explain the users and decision supported by the pipeline, the prior state, your role, and the measurable functional and nonfunctional requirements. #### Hints Give the interviewer a small set of constraints that make later trade-offs inevitable. Unsupported scale numbers are less useful than a clear workload shape and service objective. #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 2: Explain the Architecture and Trade-offs Trace one record from source through ingestion, storage, transformation, quality controls, orchestration, and serving. For each major technology or pattern, compare at least one plausible alternative. #### Hints Explain the decision criterion before naming the chosen tool. Make replay, idempotency, schema evolution, and failure isolation visible in the diagram. #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 3: Deep-Dive on a Technical Challenge Describe one concrete failure or constraint, how you diagnosed it, the options considered, the fix, validation, rollout, and measured result. Include what you would change now. #### Hints Choose a challenge whose resolution reveals engineering judgment, not merely effort. Separate the immediate mitigation from the durable correction. #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### What a Strong Answer Covers ```premium-lock What a Strong Answer Covers ``` ### Follow-up Questions 1. Which component fails first if throughput grows by an order of magnitude? 2. How do you know a successful job produced correct data? 3. How do live processing and backfills avoid overwriting each other? 4. What was the most expensive design choice, and how did you justify it? 5. Which decision would you reverse if freshness requirements became much looser?