Netflix Ads Feature Launch: Staged Release, Holdouts and Two-Sided Success Metrics

You are a product manager at Netflix. The company is getting ready to launch a new feature that earns money through advertising, and it has not yet reached members. Explain how you would release the feature and how you would test what effect it has. ```hint Read both the fast and the slow effects An ad feature can raise revenue in its first week while any harm to viewing or retention shows up only months later. Decide which effects you must measure quickly, which need a longer window, and how the rollout keeps a way to see both. ``` ### Clarifying Questions - What exactly is the feature: a new ad format or placement shown to members already on the plan with ads, a new ad-supported way to watch for people who see no ads today, or something else? - Which outcome matters most to the business: more advertising revenue per member, more members choosing the plan with ads, or total revenue including subscriptions? - Will it launch only in markets where the plan with ads already exists, and do advertising deals already sold for the coming months limit how much inventory a test can hold back? - Which devices must support it at launch: TVs, phones, tablets, the web, or only some of them? ### What a Strong Answer Covers - A staged rollout with entry and exit criteria for each stage, a kill switch, and a reason for which markets and devices go first - A success metric tied to the business goal, with member guardrails such as viewing time, cancellations and moves between plans - A sound experiment: the right unit of randomization, a long-running holdout for slow effects, and how shared advertiser budgets can bias a revenue comparison - The advertiser side: delivery, measurement, and the commitments that limit how much inventory can be withheld - A clear rule for expanding, iterating or pulling the feature back ### Follow-up Questions - After two weeks, advertising revenue per member is up and viewing hours per member are slightly down, with no change yet in cancellations. What do you recommend? - How would you tell whether the feature pushes members toward an ad-free plan or out of Netflix entirely, and how would you weigh each outcome? - Sales wants the feature live in every market at once for a large advertiser campaign. How do you respond?