Retention Model Pitfalls: Multicollinearity, Causal Feature Effects, Feature Importance

A consumer rewards app wants to understand user retention. In a statistics round you are given a modeling scenario and asked what to watch out for when modeling, how to deal with multicollinearity and whether it affects the results, how to determine whether a particular feature improves retention, and how to read feature importance. The report does not describe the scenario's data. Assume one row per user, a binary label for whether the user was retained (under a definition the team has already chosen), and a few dozen candidate features describing each user's activity and profile, several of which are strongly correlated with each other, such as the number of sessions and the number of offers the user redeemed. ### Clarifying Questions - How is retention defined, and over what period are the features measured relative to the retention window? - Is the main goal to predict which users will churn, or to explain what drives retention? - Is the feature the team asks about something the product can change, such as a capability users can turn on, or a characteristic of the user? ### Part 1 — What to watch out for What would you check and watch out for when building this retention model? ```hint Timing of features For each feature, ask whether its value could already reflect whether the user was retained. ``` #### What This Part Should Cover - The label definition and leakage from features measured during or after the retention window - A validation scheme that matches how the model will be used - Data issues specific to retention: class balance, users with little history, and missing values that carry meaning ### Part 2 — Multicollinearity What is multicollinearity, how would you detect it in this data, does it affect the results, and what would you do about it? ```hint Which results? Separate what the model predicts from what it says about each individual feature. ``` #### What This Part Should Cover - How to detect it in this data - Its effect on coefficient estimates, standard errors and interpretation, compared with its effect on predictions - Remedies matched to the goal of the model, and what each costs ### Part 3 — Does a feature improve retention? The team wants to know whether a specific feature improves retention. How would you determine that? ```hint What the coefficient can and cannot say Think about why users who have this feature might have been more likely to stay anyway. ``` #### What This Part Should Cover - Why an association in the model does not establish an effect - The preferred design when the product can change the feature, and alternatives when it cannot - The assumptions behind each design and how to check them ### Part 4 — Feature importance How would you read feature importance from the model, and what are its limits? ```hint Correlated features again Think about what happens to an importance score when two features carry nearly the same information. ``` #### What This Part Should Cover - The main importance measures and what each actually measures - How correlated features distort importance - What importance does not tell you ### What a Strong Answer Covers - Distinguishes prediction, explanation and causal questions throughout, and ties each recommendation to one of them - Concrete checks for leakage, multicollinearity and importance rather than lists of terms - Connects multicollinearity to both coefficient interpretation and feature importance - A practical, defensible plan for the team's question about whether one feature improves retention ### Follow-up Questions - Two features have a correlation of about 0.95, and one of them gets a negative coefficient even though on its own it is positively associated with retention. How do you explain this to a product manager? - An experiment is not possible for the feature. Which observational design would you use, and which assumption would you have to defend? - Permutation importance and SHAP rank the top features differently. Which do you trust, and why?