Evaluate new shop-ads ranking algorithm
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: easy
Interview Round: Technical Screen
You work on a marketplace with **shop ads**. A new ranking/recommendation algorithm is proposed to **promote shop ads** more aggressively, but stakeholders are unclear on *what exactly it changes* and *how to judge whether it is better*.
Design an evaluation plan with an online experiment.
## What to cover
1. **Clarify the product change**: What questions would you ask to understand what the new algorithm is optimizing/promoting (e.g., different inventory, different bids, different relevance model)?
2. **Experiment design**:
- How would you set up an **A/B test**?
- What is the correct **randomization unit** (user vs session), and what are the tradeoffs?
- Any concerns about **interference/market-level effects** (e.g., auctions, budgets) and how you’d mitigate them.
3. **Metrics**:
- Propose **primary**, **diagnostic**, and **guardrail** metrics.
- Include at least: CTR rising but revenue falling (a metric conflict) — how would you interpret and debug it?
4. **Power / sample size**:
- Outline how you’d do **power analysis** (inputs needed, MDE, variance estimation), and what you’d do if the experiment is underpowered.
5. **Decision & follow-ups**:
- How you would segment results (e.g., by geography) and decide whether to ship, iterate, or roll back.
- Any offline evaluation you’d do before/alongside the A/B test.
State any assumptions you make (auction type, billing model CPC/CPM, budgets, etc.).
Quick Answer: This question evaluates a data scientist's skills in online experimentation, causal inference, and marketplace analytics—covering A/B test design, randomization and interference concerns, auction and budget effects, metric selection (primary, diagnostic, guardrail), and power/sample-size reasoning.