Time-Series Forecasting For Product Metrics

What's being tested

Interviewers are checking your ability to turn noisy product telemetry into reliable short-to-medium‑term forecasts that support decisions (capacity, experiments, rollouts). You'll be evaluated on metric definition, baseline selection, feature engineering for calendar/events, model choice and validation (especially rolling-origin cross‑validation), and how you quantify and communicate uncertainty and business impact. Meta cares because accurate forecasts of DAU, conversion rates, or revenue inform launch timing, capacity planning, and experiment interpretation.

Core knowledge

Metric clarity: Define numerator/denominator, deduping/windowing rules, and aggregation frequency (hourly/daily/weekly). Small definitional shifts change trends and forecast bias.
Baselines first: Implement naive (last-value), seasonal-naive, and simple exponential smoothing; short horizons often beat complex models. Always compare to these.
Decomposition: Use STL or additive/multiplicative decomposition to separate trend, seasonality, and residuals; identify weekly/weekday and monthly patterns and holiday spikes.
Stationarity & differencing: For ARIMA(p,d,q), test stationarity (ADF); choose differencing d to remove unit roots. Notation: ARIMA(p,d,q) with seasonal P,D,Q,m.
Feature engineering for ML: Create lag features, rolling aggregates (7/28-day mean), calendar flags, experiment flags, and external regressors (ads spend). For daily DAU, include lags t-1, t-7, t-14 and weekday dummies.
Cross‑validation: Use rolling‑origin (time-series) CV: expand or sliding windows with consistent train→test chronological splits. Report average metric over folds and variation.
Evaluation metrics: Prefer MASE for scale-free comparison:

$MASE = \frac{\frac{1}{T}\sum_{t=1}^T |e_t|}{\frac{1}{T-1}\sum_{t=2}^T |y_t - y_{t-1}|}$

Avoid MAPE with zeros; use RMSE for penalizing large errors and quantile loss for probabilistic forecasts.
Probabilistic forecasts: Communicate prediction intervals (e.g., 90% PI) using analytic intervals from state-space models or empirical bootstrapping; for business decisions, quantify risk thresholds.
Model classes & tradeoffs: ETS/ARIMA are interpretable for single series; global ML models (XGBoost) scale across many related series but risk data leakage; deep learning (LSTM) needs lots of data and careful validation.
Handling events and structural breaks: Encode promotions/holiday flags or use intervention analysis; after release/bug, prefer re-fitting quickly and adjust CV to avoid leakage from the shift.
Scale guidance: For hundreds of series, per-series ARIMA is feasible; for thousands+, prefer global models or automated pipelines like Prophet or light-weight state-space methods.
Business alignment: Translate forecast error into business impact (capacity cost, missed-revenue), and pick horizons aligned to decisions (e.g., 7 days for short ops, 28–90 days for planning).

Tip: always log-transform counts when variance scales with level, but validate back-transformation bias.

Worked example — "Forecast next 28-day DAU for a product feature"

Framing (first 30s): confirm the exact DAU definition, aggregation timezone, desired forecast horizon (28 days) and cadence (daily), acceptable latency, and whether holidays/promos will occur. Clarify whether you must produce point forecasts, intervals, or both.

Skeleton of a strong answer:

Establish baselines: seasonal‑naive (last 28-day pattern) and ETS/Holt‑Winters for short comparison.
Decompose with STL to visualize weekly weekday effects, trend, and anomalies; flag holiday/promotional days.
Feature engineering: create lags (1,7,14), rolling means (7,28), weekday dummies, and experiment/promo indicators as regressors.
Model selection: prefer ETS or state‑space for interpretability and uncertainty; try XGBoost with lag features as a global model if you have many related features/series.
Validation: rolling-origin CV with multiple 28-day test windows; report MASE and 90% prediction intervals.

Tradeoff to flag: a global XGBoost can leverage cross-series patterns and improve accuracy, but risks leakage if series have nonstationary regime changes; if business needs explainability, prefer interpretable state‑space models.

Close: say you'll productionize via automated retraining, monitor forecast drift, and if more time would test probabilistic calibration and integrate forecast error into decision thresholds (e.g., alerts when prediction < threshold).

A second angle — "Detect and forecast a post-release drop in conversion rate"

Here the target is a rate (conversion) with experiment flags. You'd model underlying seasonality and then estimate the counterfactual: use a forecasting model trained on pre-release history plus covariates to predict the expected conversion after release. Compare actual to forecast to estimate uplift/decline, and quantify uncertainty via prediction intervals or synthetic-control. Important differences: rates have bounded support [0,1], so consider logit transform or beta regression; include experiment indicator to avoid confounding, and use shorter CV windows around the release to capture local behavior. If the release is an A/B test, complement forecasts with causal inference (pre‑post paired tests or difference‑in‑differences) rather than pure forecasting.

Common pitfalls

Pitfall: Using MAPE blindly — When a series contains zeros or near‑zeros, MAPE explodes and misleads model selection; use MASE or scale‑aware metrics instead.

Overfitting via leakage — People include future information (holiday labels that encode future outcomes or smoothed target features computed with full-series stats). Always compute features using only historical data up to each training cutoff in CV.

Communication mistake — Presenting only point forecasts without intervals or actionability is risky. Stakeholders need the probability of breach (e.g., capacity >95th percentile) and suggested mitigations tied to forecast uncertainty.

Connections

Forecasting often pivots into anomaly detection (differences between forecast and observed), experimentation/causal inference (constructing counterfactuals), and hierarchical forecasting (aggregating forecasts across product segments). Be ready to discuss how forecasts feed metrics dashboards and experiment analyses.