How would you evaluate upranking shop ads?
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: easy
Interview Round: Technical Screen
## Context
You work on an ads platform (e.g., FB/IG). The team proposes **upranking “Shop Ads”** (ads that lead to an in-app shop/catalog checkout flow) relative to **Website Ads** (ads that click out to an advertiser’s website).
The motivation is that Shop Ads may reduce friction for users and help small advertisers who don’t have a strong website, but may hurt large advertisers optimizing for different objectives (e.g., in-store foot traffic, brand, or deep-funnel website conversions).
## Task
1. **Evaluate the idea**: What are the potential benefits, risks, and unintended consequences of upranking Shop Ads?
2. **Define success**:
- Propose a set of **primary metrics**, **diagnostic metrics**, and **guardrail metrics**.
- Explain tradeoffs (user experience vs revenue vs advertiser welfare).
3. **Design an experiment** to measure impact:
- Unit of randomization (user, auction, advertiser), treatment definition, duration.
- How you would handle interference/network effects and auction dynamics.
- How you would segment results (e.g., small vs large advertisers) and avoid Simpson’s paradox.
4. **If you observe mixed effects** (e.g., platform revenue up but large advertisers’ ROAS down), how would you interpret results and decide whether to launch, iterate, or roll back?
### Assumptions (you may modify)
- Ads are served via an auction with a ranking score.
- Shop Ads and Website Ads compete in the same auction.
- You can log impressions, clicks, purchases, and attributed conversions (with delay).
Quick Answer: This question evaluates a candidate's competency in ad-ranking intervention evaluation, causal inference and experimentation design, metric definition, and trade-off analysis within the Analytics & Experimentation domain, requiring both conceptual understanding of incentives and practical application to metric selection and randomized test design.