Pinterest Home Feed Ad Load Optimization

You are asked to design an analysis and experiment to determine whether the current home-feed ad load (ads per 100 content units) is optimized.

Assumptions and context:

• Ad load L is defined as the number of ad impressions per 100 organic content units rendered in the home feed.
• Ads are allocated via an auction; ad demand, prices, and user engagement respond to changes in supply.
• Primary stakeholders: users (experience, retention), advertisers (ROI), and the business (revenue).

Tasks

1. Primary success function and guardrails
   • Propose a business objective that trades off user experience and ads monetization (e.g., maximize revenue subject to user-experience guardrails).
   • Define exact user guardrails (e.g., 7-day retention, session duration, complaint rate) and ads-side metrics (e.g., RPM, CTR, advertiser ROI), including formulas.
2. Experiment design
   • Specify experiment arms, randomization unit, holdouts, duration, and power.
   • Provide a plan to mitigate auction interference and novelty effects.
   • Describe how to select long-term proxy metrics for user value and how to backtest their predictive validity.
3. Decision rule
   • Formalize the decision as an optimization problem with constraints (e.g., maximize RPM with hard constraints on relative changes in retention and time spent).
   • Specify the statistical testing approach (sequential vs fixed-horizon) and how you will control for multiple testing across arms and metrics.
4. Heterogeneity
   • Explain how you would segment users/content (e.g., new vs power users, geos, devices, content verticals), test for interactions, and guard against Simpson’s paradox.
5. Rollout plan
   • Provide a ramp strategy, kill switches/guardrails, and a monitoring plan post-launch for non-stationarity and seasonality.
   • Propose concrete metric formulas and thresholds you would use.