Design a robust conversion propensity model

Quick Overview

This question evaluates a candidate's competency in end-to-end propensity modeling, covering correct labeling to avoid post-treatment leakage, time-windowed feature engineering, class-imbalance and calibration strategies, causal uplift estimation, and production concerns such as monitoring, serving consistency, and cold-start scoring.

Design a robust conversion propensity model

Company: Netflix

Role: Data Scientist

Category: Machine Learning

Difficulty: hard

Interview Round: Onsite

You are the modeling DS for notifications at a marketplace. Goal: score users daily with a propensity to purchase within 7 days if sent a promo notification today; only top 20% by score will be contacted. Design the model end-to-end: (1) Labeling and leakage: define a correct label that avoids post-treatment leakage when historical notifications already influenced behavior; handle users with multiple exposures; define negative windows; decide whether to use intent-to-treat or treated-only labels and justify. (2) Features: propose time-windowed behavioral features, catalog/category signals, price sensitivity, recency/frequency, user–item interactions; specify how to avoid target leakage, enforce time-consistent joins, and mitigate training–serving skew. (3) Class imbalance and calibration: choose loss, regularization, and calibration method; explain how you will monitor and recalibrate over time. (4) Offline evaluation: pick metrics (e.g., PR-AUC for ranking, calibration error), construct time-based splits, and design slice analyses for country and tenure. (5) Causal lift and policy value: with historical logs lacking randomization, propose an approach (e.g., inverse propensity weighting or doubly robust estimation) to estimate incremental revenue of the top-20% policy; describe how you will get propensities and reduce bias (overlap checks, trimming). (6) Online validation and ramp: define guardrails and primary metrics, traffic split, holdout policy, ramp criteria, and a plan to detect feedback loops and non-stationarity. (7) Cold start: describe how you will score new users/items on day 0 and backfill training labels over time.

Quick Answer: This question evaluates a candidate's competency in end-to-end propensity modeling, covering correct labeling to avoid post-treatment leakage, time-windowed feature engineering, class-imbalance and calibration strategies, causal uplift estimation, and production concerns such as monitoring, serving consistency, and cold-start scoring.

|Home/Machine Learning/Netflix
Netflix logo
Netflix
Oct 13, 2025, 9:49 PM
hardData ScientistOnsiteMachine Learning
15
0

Daily Notification Propensity Model (Top-20% Targeting)

Context

You need to score users once per day with the probability they will make a purchase within the next 7 days if sent a promotional notification today. Only the top 20% by score will be contacted. Historical logs include notifications, user activity, catalog interactions, and purchases. Historical notifications may already have influenced behavior.

Task

Design the model end-to-end:

  1. Labeling and leakage
    • Define a correct label that avoids post-treatment leakage when historical notifications already influenced behavior.
    • Handle users with multiple exposures within an outcome window.
    • Define negative windows.
    • Decide whether to use intent-to-treat (ITT) or treated-only labels; justify your choice.
  2. Features
    • Propose time-windowed behavioral features, catalog/category signals, price sensitivity, recency/frequency, and user–item interactions.
    • Specify how to avoid target leakage, enforce time-consistent joins, and mitigate training–serving skew.
  3. Class imbalance and calibration
    • Choose loss, regularization, and calibration method.
    • Explain how you will monitor and recalibrate over time.
  4. Offline evaluation
    • Pick metrics (e.g., PR-AUC for ranking, calibration error).
    • Construct time-based splits.
    • Design slice analyses for country and tenure.
  5. Causal lift and policy value (no randomization in logs)
    • Propose an approach (e.g., inverse propensity weighting or doubly robust estimation) to estimate incremental revenue of the top-20% policy.
    • Describe how to get propensities and reduce bias (overlap checks, trimming).
  6. Online validation and ramp
    • Define guardrails and primary metrics, traffic split, holdout policy, ramp criteria.
    • Plan to detect feedback loops and non-stationarity.
  7. Cold start
    • Describe how to score new users/items on day 0.
    • Explain how to backfill training labels over time.

Assumptions

  • Daily decision time t at a fixed hour (e.g., 10:00 UTC).
  • Outcome window is [t, t+7 days).
  • Eligibility is known at t (e.g., not unsubscribed, within send limits).
  • Logs contain timestamps for views/clicks/add-to-cart/purchases/notifications.
Loading comments...