You are the modeling DS for notifications at a marketplace. Goal: score users daily with a propensity to purchase within 7 days if sent a promo notification today; only top 20% by score will be contacted. Design the model end-to-end: (1) Labeling and leakage: define a correct label that avoids post-treatment leakage when historical notifications already influenced behavior; handle users with multiple exposures; define negative windows; decide whether to use intent-to-treat or treated-only labels and justify. (2) Features: propose time-windowed behavioral features, catalog/category signals, price sensitivity, recency/frequency, user–item interactions; specify how to avoid target leakage, enforce time-consistent joins, and mitigate training–serving skew. (3) Class imbalance and calibration: choose loss, regularization, and calibration method; explain how you will monitor and recalibrate over time. (4) Offline evaluation: pick metrics (e.g., PR-AUC for ranking, calibration error), construct time-based splits, and design slice analyses for country and tenure. (5) Causal lift and policy value: with historical logs lacking randomization, propose an approach (e.g., inverse propensity weighting or doubly robust estimation) to estimate incremental revenue of the top-20% policy; describe how you will get propensities and reduce bias (overlap checks, trimming). (6) Online validation and ramp: define guardrails and primary metrics, traffic split, holdout policy, ramp criteria, and a plan to detect feedback loops and non-stationarity. (7) Cold start: describe how you will score new users/items on day 0 and backfill training labels over time.

This question evaluates a candidate's competency in end-to-end propensity modeling, covering correct labeling to avoid post-treatment leakage, time-windowed feature engineering, class-imbalance and calibration strategies, causal uplift estimation, and production concerns such as monitoring, serving consistency, and cold-start scoring.

How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

What difficulty level is this interview question?

This is a hard difficulty Machine Learning question, commonly asked during Onsite rounds at Netflix.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Netflix during technical interviews.

Design a robust conversion propensity model | Netflix Interview Question

Daily Notification Propensity Model (Top-20% Targeting)

Context

You need to score users once per day with the probability they will make a purchase within the next 7 days if sent a promotional notification today. Only the top 20% by score will be contacted. Historical logs include notifications, user activity, catalog interactions, and purchases. Historical notifications may already have influenced behavior.

Task

Design the model end-to-end:

Labeling and leakage
- Define a correct label that avoids post-treatment leakage when historical notifications already influenced behavior.
- Handle users with multiple exposures within an outcome window.
- Define negative windows.
- Decide whether to use intent-to-treat (ITT) or treated-only labels; justify your choice.
Features
- Propose time-windowed behavioral features, catalog/category signals, price sensitivity, recency/frequency, and user–item interactions.
- Specify how to avoid target leakage, enforce time-consistent joins, and mitigate training–serving skew.
Class imbalance and calibration
- Choose loss, regularization, and calibration method.
- Explain how you will monitor and recalibrate over time.
Offline evaluation
- Pick metrics (e.g., PR-AUC for ranking, calibration error).
- Construct time-based splits.
- Design slice analyses for country and tenure.
Causal lift and policy value (no randomization in logs)
- Propose an approach (e.g., inverse propensity weighting or doubly robust estimation) to estimate incremental revenue of the top-20% policy.
- Describe how to get propensities and reduce bias (overlap checks, trimming).
Online validation and ramp
- Define guardrails and primary metrics, traffic split, holdout policy, ramp criteria.
- Plan to detect feedback loops and non-stationarity.
Cold start
- Describe how to score new users/items on day 0.
- Explain how to backfill training labels over time.

Assumptions

Daily decision time t at a fixed hour (e.g., 10:00 UTC).
Outcome window is [t, t+7 days).
Eligibility is known at t (e.g., not unsubscribed, within send limits).
Logs contain timestamps for views/clicks/add-to-cart/purchases/notifications.

Daily Notification Propensity Model (Top-20% Targeting)

Context

Task

Design the model end-to-end:

Labeling and leakage
- Define a correct label that avoids post-treatment leakage when historical notifications already influenced behavior.
- Handle users with multiple exposures within an outcome window.
- Define negative windows.
- Decide whether to use intent-to-treat (ITT) or treated-only labels; justify your choice.
Features
- Propose time-windowed behavioral features, catalog/category signals, price sensitivity, recency/frequency, and user–item interactions.
- Specify how to avoid target leakage, enforce time-consistent joins, and mitigate training–serving skew.
Class imbalance and calibration
- Choose loss, regularization, and calibration method.
- Explain how you will monitor and recalibrate over time.
Offline evaluation
- Pick metrics (e.g., PR-AUC for ranking, calibration error).
- Construct time-based splits.
- Design slice analyses for country and tenure.
Causal lift and policy value (no randomization in logs)
- Propose an approach (e.g., inverse propensity weighting or doubly robust estimation) to estimate incremental revenue of the top-20% policy.
- Describe how to get propensities and reduce bias (overlap checks, trimming).
Online validation and ramp
- Define guardrails and primary metrics, traffic split, holdout policy, ramp criteria.
- Plan to detect feedback loops and non-stationarity.
Cold start
- Describe how to score new users/items on day 0.
- Explain how to backfill training labels over time.

Assumptions

Daily decision time t at a fixed hour (e.g., 10:00 UTC).
Outcome window is [t, t+7 days).
Eligibility is known at t (e.g., not unsubscribed, within send limits).
Logs contain timestamps for views/clicks/add-to-cart/purchases/notifications.

Design a robust conversion propensity model

Quick Overview

Daily Notification Propensity Model (Top-20% Targeting)

Context

Task

Assumptions

Solution

Comments (0)

Design a robust conversion propensity model

Quick Overview

Daily Notification Propensity Model (Top-20% Targeting)

Context

Task

Assumptions

Solution

Comments (0)