You inherit a churn prediction model trained to predict whether a user will place an order in the next 28 days. The training data include features computed weekly, some of which leak post-label information (e.g., features that include activity within the prediction window). (a) Identify at least five concrete leakage sources in a logs-based feature set (examples: features derived from orders within the label window, coupon_applied in the next 28 days, post-treatment delivery ETA, label-influenced support contacts). For each, rewrite the feature so it is computable at prediction time with a strict event-time cutoff. (b) Propose a time-based cross-validation scheme (rolling origin) and define train/validation/test splits ensuring no future leakage. Specify how you would handle users entering/leaving the cohort and cold-start users. (c) Offline metrics show AUC=0.79, PR-AUC=0.23. Online, targeting the top decile increases conversion by 1.5% but raises cancellations by 0.3pp. Define business KPIs and a cost-sensitive objective to tune the decision threshold; include calibration (Platt/Isotonic) and how you would check calibration drift in production. (d) Interpretation: a standardized feature past_7d_orders has logistic regression coefficient 0.40 and baseline log-odds of churn −1.50. Compute the odds ratio for a +1 SD increase and the resulting churn probability change from the baseline. Explain limitations of such ceteris paribus interpretations in correlated settings. (e) Outline a monitoring plan (data quality, feature distributions, PSI, label delay, prediction drift) and a retraining policy tied to performance and covariate shift triggers.

This question evaluates competence in production-ready supervised machine learning—detecting and repairing data leakage in logs-based features, designing time-aware cross-validation, defining KPI-driven thresholding and calibration, interpreting model coefficients, and establishing monitoring and retraining policies for a churn prediction model in the Machine Learning domain. It is commonly asked because interviewers need to assess both conceptual understanding of leakage, validation, and calibration principles and practical application skills for deploying and maintaining models with cost-sensitive objectives and monitoring in production, so the level of abstraction spans conceptual reasoning and hands-on practical application.

How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

What difficulty level is this interview question?

This is a hard difficulty Machine Learning question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Detect leakage and evaluate a prediction model

Churn Prediction Model: Leakage, Validation, KPIs, Interpretation, Monitoring

Context: You inherit a weekly-scored model that predicts whether a user will place an order in the next 28 days. Some features were built from logs in ways that leak information from the post-prediction label window. Address the following tasks.

(a) Leakage identification and repair

Identify at least five concrete leakage sources that can occur in a logs-based feature set (e.g., features derived from orders within the label window, coupon_applied in the next 28 days, post-treatment delivery ETA, label-influenced support contacts).
For each, rewrite the feature so it is computable at prediction time with a strict event-time cutoff.

(b) Time-based cross-validation (rolling origin)

Propose a rolling origin cross-validation scheme and define train/validation/test splits that ensure no future leakage.
Specify how to handle users entering/leaving the cohort across time and how to handle cold-start users.

Offline metrics are AUC = 0.79 and PR-AUC = 0.23. Online, targeting the top decile increases conversion by 1.5% but raises cancellations by 0.3pp.
Define business KPIs and a cost-sensitive objective to tune the decision threshold.
Include probability calibration (Platt or Isotonic) and how you would check calibration drift in production.

(d) Interpretation

A standardized feature past_7d_orders has a logistic regression coefficient of 0.40 and a baseline log-odds of churn of −1.50.
Compute the odds ratio for a +1 SD increase and the resulting change in churn probability from the baseline.
Explain the limitations of such ceteris paribus interpretations in correlated feature settings.

(e) Monitoring and retraining

Outline a monitoring plan (data quality, feature distributions, PSI, label delay, prediction drift) and a retraining policy tied to performance and covariate shift triggers.

Churn Prediction Model: Leakage, Validation, KPIs, Interpretation, Monitoring

(a) Leakage identification and repair

Identify at least five concrete leakage sources that can occur in a logs-based feature set (e.g., features derived from orders within the label window, coupon_applied in the next 28 days, post-treatment delivery ETA, label-influenced support contacts).
For each, rewrite the feature so it is computable at prediction time with a strict event-time cutoff.

(b) Time-based cross-validation (rolling origin)

Propose a rolling origin cross-validation scheme and define train/validation/test splits that ensure no future leakage.
Specify how to handle users entering/leaving the cohort across time and how to handle cold-start users.

Offline metrics are AUC = 0.79 and PR-AUC = 0.23. Online, targeting the top decile increases conversion by 1.5% but raises cancellations by 0.3pp.
Define business KPIs and a cost-sensitive objective to tune the decision threshold.
Include probability calibration (Platt or Isotonic) and how you would check calibration drift in production.

(d) Interpretation

A standardized feature past_7d_orders has a logistic regression coefficient of 0.40 and a baseline log-odds of churn of −1.50.
Compute the odds ratio for a +1 SD increase and the resulting change in churn probability from the baseline.
Explain the limitations of such ceteris paribus interpretations in correlated feature settings.

(e) Monitoring and retraining

Outline a monitoring plan (data quality, feature distributions, PSI, label delay, prediction drift) and a retraining policy tied to performance and covariate shift triggers.

Detect leakage and evaluate a prediction model

Quick Overview

Churn Prediction Model: Leakage, Validation, KPIs, Interpretation, Monitoring

Solution

Comments (0)

Detect leakage and evaluate a prediction model

Quick Overview

Churn Prediction Model: Leakage, Validation, KPIs, Interpretation, Monitoring

Solution

Comments (0)