How would you measure billboard impact?
Company: Airwallex
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
Pinterest is considering launching a new **billboard**, a large and prominent promotional module shown near the top of the home feed or another major surface to highlight a campaign, creator, topic, or commercial placement.
As a Data Scientist, design how you would evaluate whether this billboard is effective. Your answer should address:
1. **Product design and targeting**
- What user problem or business goal is the billboard solving?
- Which users should see it?
- How often should it be shown?
2. **Experiment design**
- What should the treatment and control groups be?
- What is the correct randomization unit: user, session, device, geo, or something else?
- How would you account for interference, repeated exposure, or novelty effects?
3. **Metrics**
- Define primary success metrics, secondary metrics, and guardrail metrics.
- Consider tradeoffs across short-term engagement, long-term retention, monetization, content diversity, user satisfaction, and downstream creator or advertiser outcomes.
4. **Causal inference if randomization is not feasible**
- Explain the difference between **Propensity Score Matching (PSM)** and **Propensity Score Weighting (PSW / inverse-propensity weighting)**.
- What estimand does each approach usually target, such as ATE or ATT?
- What is each method trying to optimize or balance?
- What diagnostics would you use to assess whether the method is credible?
5. **Sample size and power**
- How would you calculate the required sample size or minimum detectable effect for the main metric?
- What tradeoffs matter when choosing duration, traffic allocation, and metric sensitivity?
6. **Risks and edge cases**
- Discuss selection bias, heterogeneous treatment effects, spillovers, Simpson's paradox across segments, and other practical pitfalls.
Provide a rigorous measurement plan, not just a list of metrics.
Quick Answer: This question evaluates a data scientist's skills in experiment and product design, causal inference (including propensity-score approaches), metric definition and trade-off analysis, power and sample-size estimation, and handling biases such as interference, repeated exposure, and heterogeneous effects.