How would you evaluate upranking shop ads?
Quick Overview
This question evaluates a candidate's competency in ad-ranking intervention evaluation, causal inference and experimentation design, metric definition, and trade-off analysis within the Analytics & Experimentation domain, requiring both conceptual understanding of incentives and practical application to metric selection and randomized test design.
Context
You work on an ads platform (e.g., FB/IG). The team proposes upranking “Shop Ads” (ads that lead to an in-app shop/catalog checkout flow) relative to Website Ads (ads that click out to an advertiser’s website).
The motivation is that Shop Ads may reduce friction for users and help small advertisers who don’t have a strong website, but may hurt large advertisers optimizing for different objectives (e.g., in-store foot traffic, brand, or deep-funnel website conversions).
Task
- Evaluate the idea : What are the potential benefits, risks, and unintended consequences of upranking Shop Ads?
-
Define success
:
- Propose a set of primary metrics , diagnostic metrics , and guardrail metrics .
- Explain tradeoffs (user experience vs revenue vs advertiser welfare).
-
Design an experiment
to measure impact:
- Unit of randomization (user, auction, advertiser), treatment definition, duration.
- How you would handle interference/network effects and auction dynamics.
- How you would segment results (e.g., small vs large advertisers) and avoid Simpson’s paradox.
- If you observe mixed effects (e.g., platform revenue up but large advertisers’ ROAS down), how would you interpret results and decide whether to launch, iterate, or roll back?
Assumptions (you may modify)
- Ads are served via an auction with a ranking score.
- Shop Ads and Website Ads compete in the same auction.
- You can log impressions, clicks, purchases, and attributed conversions (with delay).