Conversion Rate Forecasting

What's being tested

Candidates must show they can turn historical conversion-rate signals into robust, business-actionable forecasts: define the metric, choose an appropriate statistical or ML model given exposure and sparsity, evaluate time-aware performance, and communicate uncertainty and tradeoffs. Interviewers probe model assumptions, handling of heteroskedasticity from varying impression counts, ways to pool information across campaigns, and how forecasts would be used to make decisions (budgeting, creative cadence, capacity).

Core knowledge

• Conversion rate formulation: conversions C over exposures/impressions N, so $p = C / N$. Treat raw counts as Binomial: $C \sim \text{Binomial}(N, p)$ whenever N is known and trials are independent.

• Rate modeling techniques: use a Binomial/Logistic GLM ($\text{logit}(p) = X\beta$) for direct rate modeling, or model counts with a Poisson/Negative Binomial with log(exposure) as an offset: $\log(\lambda) = X\beta + \log(N)$.

• Variance heteroskedasticity: variance of $p$ is $p(1−p)/N$; small N yields high variance. Weighting by N or modeling counts rather than rates avoids misleading fits.

• Link functions & transforms: apply logit: $\text{logit}(p) = \log(p/(1−p))$ for unconstrained prediction, or use Beta regression when modeling continuous rates in (0,1) with precision parameter $\phi$.

• Hierarchical / partial pooling: for many campaigns with limited history, use mixed-effects models or Bayesian hierarchical models to share signal across campaigns and avoid overfitting single-campaign noise.

• Time structure & seasonality: include lag features, rolling averages, weekly/holiday indicators, and consider state-space / ARIMA or `Prophet` for seasonality and trend components when behavior is temporal.

• Evaluation & CV: use time-series cross-validation (rolling-origin) and metrics robust to rate scale: MAE, RMSE on probabilities, calibration/Brier score, and coverage of prediction intervals; avoid random CV that leaks future info.

• Uncertainty quantification: provide prediction intervals (e.g., binomial CI, bootstrap, Bayesian posterior predictive intervals, or quantile regression); communicate heteroskedastic uncertainty driven by N.

• Cold-start / sparse campaigns: backfill with cohort-level averages, metadata-based similarity (channel, creative type), or use meta-learning / embedding features so new campaigns borrow strength.

• Feature engineering: include spend, bids, placement, creative attributes, prior-period clicks, device mix, price, inventory, and external signals (holidays). Interaction terms often matter (promotion × channel).

• Practical scale/tradeoffs: per-campaign Bayesian hierarchical models work well for thousands of groups; for tens of millions of rows, use regularized tree models like `LightGBM` or aggregated GLMs via `statsmodels`/`scikit-learn`. If N ~ 10M+ time series, pre-aggregate to daily or weekly per campaign.

• Model selection & business objective: define whether you optimize point forecasts (MAE) or tail risk (quantiles) or ranking (AUC for high/low converters). Choose objective aligned to downstream use (budget allocation vs. alerting).

• Bias from promotions & experiments: flagged promotions or ongoing A/B tests confound historical patterns; include experiment assignment as covariate or exclude those windows.

Worked example — Predict Next-Period Conversion Rate Using Historical Campaign Data

First 30s framing: ask the aggregation level (per campaign per day?), forecast horizon (next day, week), definition of conversions and exposures, available covariates (spend, creatives, promotions), and what decisions will use the forecast. Skeleton answer pillars: (1) exploratory baseline — compute weighted rolling mean and binomial CIs to establish naive accuracy; (2) statistical models — fit a Binomial GLM (logit link) or Poisson with exposure offset including lags, seasonality and promotion flags; (3) hierarchical pooling — if many low-volume campaigns, fit random intercepts by campaign to shrink noisy estimates; (4) ML enhancements — a gradient-boosted tree (e.g., `XGBoost`/`LightGBM`) on campaign features for nonlinearity, but with sample weighting by N. Flag one tradeoff explicitly: per-campaign models capture idiosyncratic trends but overfit when N is small; hierarchical models reduce variance but may underfit outlier campaigns. Close with deployment-readiness: backtest with rolling-origin CV, report prediction intervals and calibration; "if I had more time, I'd incorporate uplift/campaign-treatment indicators to separate organic conversion from campaign effect and tune hierarchical priors."

A second angle

Forecasting conversion for a brand-new campaign (no history) forces different choices: rely on meta-features (audience, creative type, channel) and cohort averages, plus a hierarchical model where the campaign-level effect is drawn from a group distribution estimated from historical campaigns. You might train a model to predict campaign-level baseline conversion from metadata (a regression or embedding + `LightGBM`) and then update quickly with Bayesian online updating as early impressions arrive. This variant emphasizes transfer learning and uncertainty (wide initial intervals) more than time-series smoothing.

Common pitfalls

Pitfall: Treating rates as homoskedastic and optimizing MSE on $p$ without accounting for exposure. This downweights high-volume campaigns and leads to overconfident predictions on low-N campaigns. Instead, model counts or weight by N.

Pitfall: Showing only point forecasts to stakeholders. Business users need prediction intervals — especially for low-volume campaigns where intervals can be orders of magnitude wider.

Pitfall: Using standard random CV. Time leakage yields over-optimistic error estimates; always use rolling-origin or blocked CV matching production forecasting cadence.

Connections

This topic connects directly to A/B testing / experimentation (you must separate organic baseline vs. treatment effects) and to uplift modeling (predict incremental conversions from a campaign). Interviewers may pivot to model monitoring: drift detection on conversion distributions or business-ROI evaluation.

Further reading

• [Bayesian Data Analysis — Gelman et al.] — authoritative treatment of hierarchical models and partial pooling for sparse groups.

• Prophet (Taylor & Letham) — pragmatic time-series trend/seasonality model often used for business forecasting.