How would you measure billboard impact?

Pinterest is considering launching a new **billboard**, a large and prominent promotional module shown near the top of the home feed or another major surface to highlight a campaign, creator, topic, or commercial placement. As a Data Scientist, design how you would evaluate whether this billboard is effective. Your answer should address: 1. **Product design and targeting** - What user problem or business goal is the billboard solving? - Which users should see it? - How often should it be shown? 2. **Experiment design** - What should the treatment and control groups be? - What is the correct randomization unit: user, session, device, geo, or something else? - How would you account for interference, repeated exposure, or novelty effects? 3. **Metrics** - Define primary success metrics, secondary metrics, and guardrail metrics. - Consider tradeoffs across short-term engagement, long-term retention, monetization, content diversity, user satisfaction, and downstream creator or advertiser outcomes. 4. **Causal inference if randomization is not feasible** - Explain the difference between **Propensity Score Matching (PSM)** and **Propensity Score Weighting (PSW / inverse-propensity weighting)**. - What estimand does each approach usually target, such as ATE or ATT? - What is each method trying to optimize or balance? - What diagnostics would you use to assess whether the method is credible? 5. **Sample size and power** - How would you calculate the required sample size or minimum detectable effect for the main metric? - What tradeoffs matter when choosing duration, traffic allocation, and metric sensitivity? 6. **Risks and edge cases** - Discuss selection bias, heterogeneous treatment effects, spillovers, Simpson's paradox across segments, and other practical pitfalls. Provide a rigorous measurement plan, not just a list of metrics.