Daily Notification Propensity Model (Top-20% Targeting)

Context

You need to score users once per day with the probability they will make a purchase within the next 7 days if sent a promotional notification today. Only the top 20% by score will be contacted. Historical logs include notifications, user activity, catalog interactions, and purchases. Historical notifications may already have influenced behavior.

Task

Design the model end-to-end:

1. Labeling and leakage
   • Define a correct label that avoids post-treatment leakage when historical notifications already influenced behavior.
   • Handle users with multiple exposures within an outcome window.
   • Define negative windows.
   • Decide whether to use intent-to-treat (ITT) or treated-only labels; justify your choice.
2. Features
   • Propose time-windowed behavioral features, catalog/category signals, price sensitivity, recency/frequency, and user–item interactions.
   • Specify how to avoid target leakage, enforce time-consistent joins, and mitigate training–serving skew.
3. Class imbalance and calibration
   • Choose loss, regularization, and calibration method.
   • Explain how you will monitor and recalibrate over time.
4. Offline evaluation
   • Pick metrics (e.g., PR-AUC for ranking, calibration error).
   • Construct time-based splits.
   • Design slice analyses for country and tenure.
5. Causal lift and policy value (no randomization in logs)
   • Propose an approach (e.g., inverse propensity weighting or doubly robust estimation) to estimate incremental revenue of the top-20% policy.
   • Describe how to get propensities and reduce bias (overlap checks, trimming).
6. Online validation and ramp
   • Define guardrails and primary metrics, traffic split, holdout policy, ramp criteria.
   • Plan to detect feedback loops and non-stationarity.
7. Cold start
   • Describe how to score new users/items on day 0.
   • Explain how to backfill training labels over time.

Assumptions

• Daily decision time t at a fixed hour (e.g., 10:00 UTC).
• Outcome window is [t, t+7 days).
• Eligibility is known at t (e.g., not unsubscribed, within send limits).
• Logs contain timestamps for views/clicks/add-to-cart/purchases/notifications.