PracHub
QuestionsCoachesLearningGuidesInterview Prep
|Home/Machine Learning/Netflix

Design a robust conversion propensity model

Last updated: May 10, 2026

Quick Overview

This question evaluates a candidate's competency in end-to-end propensity modeling, covering correct labeling to avoid post-treatment leakage, time-windowed feature engineering, class-imbalance and calibration strategies, causal uplift estimation, and production concerns such as monitoring, serving consistency, and cold-start scoring.

  • hard
  • Netflix
  • Machine Learning
  • Data Scientist

Design a robust conversion propensity model

Company: Netflix

Role: Data Scientist

Category: Machine Learning

Difficulty: hard

Interview Round: Onsite

You are the modeling DS for notifications at a marketplace. Goal: score users daily with a propensity to purchase within 7 days if sent a promo notification today; only top 20% by score will be contacted. Design the model end-to-end: (1) Labeling and leakage: define a correct label that avoids post-treatment leakage when historical notifications already influenced behavior; handle users with multiple exposures; define negative windows; decide whether to use intent-to-treat or treated-only labels and justify. (2) Features: propose time-windowed behavioral features, catalog/category signals, price sensitivity, recency/frequency, user–item interactions; specify how to avoid target leakage, enforce time-consistent joins, and mitigate training–serving skew. (3) Class imbalance and calibration: choose loss, regularization, and calibration method; explain how you will monitor and recalibrate over time. (4) Offline evaluation: pick metrics (e.g., PR-AUC for ranking, calibration error), construct time-based splits, and design slice analyses for country and tenure. (5) Causal lift and policy value: with historical logs lacking randomization, propose an approach (e.g., inverse propensity weighting or doubly robust estimation) to estimate incremental revenue of the top-20% policy; describe how you will get propensities and reduce bias (overlap checks, trimming). (6) Online validation and ramp: define guardrails and primary metrics, traffic split, holdout policy, ramp criteria, and a plan to detect feedback loops and non-stationarity. (7) Cold start: describe how you will score new users/items on day 0 and backfill training labels over time.

Quick Answer: This question evaluates a candidate's competency in end-to-end propensity modeling, covering correct labeling to avoid post-treatment leakage, time-windowed feature engineering, class-imbalance and calibration strategies, causal uplift estimation, and production concerns such as monitoring, serving consistency, and cold-start scoring.

Related Interview Questions

  • Compare Losses and Explain LoRA - Netflix (medium)
  • Explain self-attention, LoRA, Adam vs SGD, ViT - Netflix (medium)
  • Explain tokenization and Transformer variants - Netflix (medium)
  • Design Real-Time Fraud Detection with XGBoost Model - Netflix (medium)
  • Address Fraud Detection with Imbalance and Concept Drift Solutions - Netflix (medium)
|Home/Machine Learning/Netflix

Design a robust conversion propensity model

Netflix logo
Netflix
Oct 13, 2025, 9:49 PM
hardData ScientistOnsiteMachine Learning
9
0

Daily Notification Propensity Model (Top-20% Targeting)

Context

You need to score users once per day with the probability they will make a purchase within the next 7 days if sent a promotional notification today. Only the top 20% by score will be contacted. Historical logs include notifications, user activity, catalog interactions, and purchases. Historical notifications may already have influenced behavior.

Task

Design the model end-to-end:

  1. Labeling and leakage
    • Define a correct label that avoids post-treatment leakage when historical notifications already influenced behavior.
    • Handle users with multiple exposures within an outcome window.
    • Define negative windows.
    • Decide whether to use intent-to-treat (ITT) or treated-only labels; justify your choice.
  2. Features
    • Propose time-windowed behavioral features, catalog/category signals, price sensitivity, recency/frequency, and user–item interactions.
    • Specify how to avoid target leakage, enforce time-consistent joins, and mitigate training–serving skew.
  3. Class imbalance and calibration
    • Choose loss, regularization, and calibration method.
    • Explain how you will monitor and recalibrate over time.
  4. Offline evaluation
    • Pick metrics (e.g., PR-AUC for ranking, calibration error).
    • Construct time-based splits.
    • Design slice analyses for country and tenure.
  5. Causal lift and policy value (no randomization in logs)
    • Propose an approach (e.g., inverse propensity weighting or doubly robust estimation) to estimate incremental revenue of the top-20% policy.
    • Describe how to get propensities and reduce bias (overlap checks, trimming).
  6. Online validation and ramp
    • Define guardrails and primary metrics, traffic split, holdout policy, ramp criteria.
    • Plan to detect feedback loops and non-stationarity.
  7. Cold start
    • Describe how to score new users/items on day 0.
    • Explain how to backfill training labels over time.

Assumptions

  • Daily decision time t at a fixed hour (e.g., 10:00 UTC).
  • Outcome window is [t, t+7 days).
  • Eligibility is known at t (e.g., not unsubscribed, within send limits).
  • Logs contain timestamps for views/clicks/add-to-cart/purchases/notifications.
Loading comments...

Browse More Questions

More Machine Learning•More Netflix•More Data Scientist•Netflix Data Scientist•Netflix Machine Learning•Data Scientist Machine Learning

Write your answer

Your first approved answer each day earns 20 XP.

Sign in to write your answer.
PracHub

Master your tech interviews with 8,000+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • AI Coding Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.