Evaluate and Monitor a Paid Electricity Price Forecast

## Interview Prompt An electricity-market forecasting vendor offers next-week daily price forecasts for an annual subscription fee of \$75,000. You have five years of the vendor's historical predictions, realized prices, and public daily inputs such as weather and policy indicators. Design an evaluation that decides whether the forecast adds economic value over a transparent baseline. Then explain how you would diagnose five consecutive days of degraded accuracy after purchase. ### Constraints & Assumptions - Electricity cannot be economically stored in this simplified market and traders must close positions before delivery. - Daily consumption is assumed not to react to within-day price changes. - Temporal leakage is forbidden; only information available before each forecast may be used. - Value must be evaluated for a named user, such as a utility risk manager or a directional trader. ### Clarifying Questions to Ask - What exact target and forecast horizon does the vendor provide? - Which loss function maps forecast error to the user's actual decision cost? - Were historical vendor forecasts generated in real time or reconstructed later? ### What a Strong Answer Covers - A time-based backtest against persistence, seasonal, and public-data baselines. - Metrics matched to use case: directional accuracy, MAE/RMSE, calibration, tail loss, or economic utility. - Uncertainty, transaction costs, position constraints, and subscription cost in the decision rule. - Statistical comparison across seasons and regimes rather than one aggregate score. - A five-day incident diagnosis covering data quality, drift, regime change, pipeline failure, and expected variance. ### Follow-up Questions - How would a utility's asymmetric cost of under-forecasting demand change the metric? - What evidence would justify pausing the vendor signal after five bad days? - How would you combine the vendor forecast with your baseline rather than choose only one?