Design a Daily Weather-Forecast Pipeline That Runs Power Models for Energy Traders

Design a system that turns a daily weather forecast into model forecasts for a desk that trades US power and gas. The round was an architecture discussion: no code was required, only an architecture diagram or flowchart with some pseudocode, and the interviewer then drilled into each layer of the design. Every day at around 6 AM, a weather forecast is published as a CSV file at a public URL. The file contains: - 10 weather scenarios; - the next 15 days at hourly frequency, so about 360 timestamps per scenario; - weather variables such as temperature, wind, soil moisture, and cloud cover or rainfall. A quant research team already provides a set of black-box models. Four of them take the weather inputs, and their outputs feed an electricity price model: ```text weather inputs +--> Electricity Demand Model --+ +--> Wind Power Model ----------+ +--> Solar Power Model ---------+--> Electricity Price Model +--> Hydro Power Model ---------+ ``` The system must cover the whole path: daily forecast ingestion, validation, historical storage, running every model across all scenarios, aggregating the results, and making the forecasts available to traders in a dashboard. Results should be ready as soon as possible after a new forecast is published. ### Constraints and Clarifications - Given: one CSV per day at a public URL, published at around 6 AM rather than at an exact time; 10 scenarios of about 360 hourly timestamps each; the model dependency graph above; the models are black boxes owned by the research team. - Deliverable: an architecture diagram or flowchart plus pseudocode for the key steps, not working code. - Not given: the file size, whether the forecast covers one location or many, how long each model takes to run, a hard deadline for results, how many traders use the dashboard, and how long history must be kept. Ask, or state your assumptions. ### Clarifying Questions - Does the URL always serve only the latest file, and how can you tell a new day's file from yesterday's: a date in the file name, an issue time inside the file, HTTP headers? - How late can the file be in the worst case, and can the provider publish a corrected file later the same day? - How are the models packaged (Python libraries, containers, services), how long does one run take, and is each scenario independent of the others? - Does the price model need anything besides the four upstream outputs of the same scenario, such as the weather inputs themselves? - By what time do traders need the numbers, and which views do they look at first? - Which orchestration, compute and storage tools does the team already run? ### Part 1 — Ingestion, scheduling and storage What stack would you use for the ingestion service? The file arrives around 6 AM rather than exactly at 6 AM: how would you schedule the ingestion? Would you store the CSV exactly as received or transform it, and what would validation check before any model runs? ```hint A response from the URL is not necessarily today's file Think about what a fetch at exactly 6:00 can return, how you would know that the file you fetched is the new forecast and is complete, and what the job should do while it is not. ``` #### What This Part Should Cover - An ingestion stack and a trigger that copes with a variable publish time - Detecting a new, complete file, and making repeated fetches idempotent - The raw-versus-transformed storage decision and the history it enables - Concrete validation checks, and which failures block the model runs ### Part 2 — Orchestration and model runs Would you build this as one DAG or as several, and how would you split the work into tasks? How do you kick off the model runs: if you orchestrate with Airflow, are they part of the same workflow as ingestion? Sketch the task graph across all models and all scenarios. ```hint Follow the dependency graph per scenario Look at which model runs depend on which, and ask whether the price model for one scenario has to wait for anything from another scenario. ``` #### What This Part Should Cover - The DAG boundaries and the reasons for them, such as reruns, backfills and ownership - How model runs are triggered and parameterized by forecast and scenario - Where the black-box models execute and how their versions are pinned - The critical path from publication to results ### Part 3 — Failure handling Where could this pipeline fail, and how would you handle the different kinds of failure? ```hint Walk the path once Go stage by stage, from the provider's website to the trader's screen, and for each stage ask what traders should see if it breaks. ``` #### What This Part Should Cover - Failures at each stage: the source file, validation, model runs, storage, orchestration and the dashboard - Which failures are retried automatically and which page a person, and how retries stay safe - What traders see when a forecast is late, partial or wrong ### Part 4 — Summary aggregation and the trader dashboard How would you aggregate the model outputs into summaries, and how would you actually make the forecasts available to traders? Which dashboard stack would you use, and how would you host it? ```hint Ten scenarios form a distribution Decide what a trader needs from 10 versions of the same hourly series, and how a new forecast should be compared with the previous one. ``` #### What This Part Should Cover - The summary statistics and time aggregations, and where they are precomputed - A dashboard stack and hosting choice, with its trade-offs - How a new day's results go live without traders seeing a half-updated mix of two forecasts ### Part 5 — Hundreds or thousands of scenarios There are 10 scenarios today. If there were hundreds or even thousands of scenarios in the future, what would you change? ```hint Count the runs Multiply out the number of model runs and output rows per forecast, then ask which part of your design breaks first. ``` #### What This Part Should Cover - Which components stop scaling first, and why - How model execution and storage change - How aggregation and the dashboard change when scenarios are numerous ### What a Strong Answer Covers - Asks how to identify the new file, how late it can be, how long the models run and when traders need results, before designing - An end-to-end diagram from the public URL to the dashboard with clear stage boundaries - Stages that are idempotent and re-runnable, keyed by forecast issue and scenario - Parallel model execution that follows the dependency graph, with the critical path identified - Failure handling and monitoring tied to what traders see - Scaling changes derived from the run count rather than generic advice ### Follow-up Questions - The provider publishes a corrected file later the same morning, after your models have already run. What happens to the results and to the dashboard? - The research team ships a new version of the wind model. How do you roll it out, and how do you rerun it on last month's forecasts to compare it with the old version? - Traders want to compare today's price forecast for next Thursday with what the models predicted for the same hours a week ago. What must you store to support that? - One of the models is far slower than the others. How does that change the pipeline and what traders see?