Optimize Email Strategy for New Prime Video Series Launch

Quick Overview

Evaluates email targeting and ranking strategy for a new Prime Video series launch. Strong answers cover data needs, cold start, ranking models, offline metrics, A/B testing, guardrails, and model selection.

Optimize Email Strategy for New Prime Video Series Launch

Company: Amazon

Role: Data Scientist

Category: Machine Learning

Difficulty: medium

Interview Round: Onsite

##### Scenario Designing, deploying and evaluating ranking models and marketing emails for Prime Video ##### Question How would you approach sending marketing emails to customers to introduce a new Prime Video series? Outline your first steps, data requirements, and modeling approach. If the ranking function changes, how would you test whether the new function performs better? When you have several ranking functions, how would you determine which one is best? ##### Hints Discuss experiment design, offline evaluation metrics (e.g., NDCG, MAP), online A/B testing, statistical significance, and user engagement KPIs.

Quick Answer: Evaluates email targeting and ranking strategy for a new Prime Video series launch. Strong answers cover data needs, cold start, ranking models, offline metrics, A/B testing, guardrails, and model selection.

|Home/Machine Learning/Amazon
Amazon logo
Amazon
Jul 12, 2025, 6:59 PM
mediumData ScientistOnsiteMachine Learning
69
0

Optimizing Email Strategy for a New Prime Video Series Launch

You are designing, deploying, and evaluating ranking models and marketing emails for Prime Video. The goal is to introduce customers to a new series while protecting user experience and email deliverability.

Outline your first steps, data requirements, modeling approach, and evaluation plan. Explain how you would test a changed ranking function and compare several ranking functions.

Constraints & Assumptions

  • Assume customers have appropriate marketing permissions and send caps apply.
  • The new series may have limited watch history, so cold-start handling matters.
  • Separate targeting, ranking, creative, send timing, and frequency.
  • Include both offline ranking evaluation and online incremental impact.

Clarifying Questions to Ask Guidance

  • What is the launch goal: episode starts, watch hours, completion, retention, or subscriptions?
  • Which users are eligible, and what compliance or unsubscribe constraints exist?
  • What candidate content and creative variants can appear in the email?
  • What historical data exists for similar series, genres, actors, or user viewing behavior?

Part 1 - First Steps and Data Requirements

Describe how you would frame the problem and gather data.

What This Part Should Cover Guidance

  • Define primary objective, guardrails, target population, attribution window, and send policy.
  • Use viewing history, genre affinity, search/browse behavior, watchlist, ratings, demographics where allowed, device, locale, and email engagement.
  • Include series metadata, similar titles, creative assets, and availability.
  • Handle opt-outs, deliverability, frequency caps, and cold start.

Part 2 - Modeling and Ranking Approach

Design targeting and ranking models for the campaign.

What This Part Should Cover Guidance

  • Use candidate generation, propensity or uplift models, collaborative filtering, content-based features, or hybrid recommenders.
  • Rank by expected incremental value subject to user experience and frequency constraints.
  • Include calibration, exploration, diversity, and fairness of exposure where relevant.
  • Consider business rules for launch priorities and cold-start metadata.

Part 3 - Testing a New Ranking Function

If the ranking function changes, how would you test whether the new function performs better?

What This Part Should Cover Guidance

  • Use offline metrics such as NDCG, MAP, precision/recall at k, calibration, and replay checks.
  • Run online A/B tests with user-level randomization and consistent send policy.
  • Measure incremental plays, watch hours, retention, unsubscribes, spam complaints, email opens/clicks, and deliverability.
  • Predefine statistical tests, sample size, and guardrail thresholds.

Part 4 - Choosing Among Several Ranking Functions

When you have several ranking functions, how would you determine which one is best?

What This Part Should Cover Guidance

  • Use offline screening to eliminate weak candidates.
  • Use multi-arm experiments, sequential testing, or bandits carefully if exploration is acceptable.
  • Compare primary metrics, guardrails, segments, and long-term effects.
  • Choose the model with the best incremental value under constraints, not only the highest offline score.

Follow-up Questions Guidance

  • How would you handle users who never open marketing emails?
  • What if the model increases clicks but also increases unsubscribes?
  • How would you evaluate a new series with no prior viewing data?
Loading comments...

Browse More Questions

More Machine Learning•More Amazon•More Data Scientist•Amazon Data Scientist•Amazon Machine Learning•Data Scientist Machine Learning