Forecast response-rate trends with backtesting
Overview
This question evaluates proficiency in time-series forecasting and model validation, including feature engineering, model selection, rolling-origin backtesting, evaluation metrics, and change-point detection for response-rate prediction at the job_category-week level.
Forecasting Response Rate by Job Category and Week
Context
You are given weekly marketplace data with invitations and responses by job_category and region. Define response rate at the job_category-week level as:
- response_rate = responses / invitations
Assume invitations are the denominator of interest and weeks without invitations are excluded or imputed carefully. The goal is to forecast the next 4 weeks of response rates for each job_category and describe the modeling approach.
Task
Aggregate to job_category-week and build a model to predict the next-4-week response rates for each job_category. Describe the following:
- Feature set: calendar dummies, region mix, invitations per job, seasonality terms, holiday flags.
- Candidate models and why: SARIMAX with exogenous regressors vs. gradient boosted trees on lag features.
- Backtesting protocol: rolling-origin evaluation with a 4-week horizon.
- Metrics: sMAPE for accuracy, calibration of predictive intervals.
- Change-point detection to guard against structural breaks, such as policy changes.
- How to use the forecasts for staffing or budget decisions.
Solution
Login required