Evaluate new shop-ads ranking algorithm
Overview
This question evaluates a data scientist's skills in online experimentation, causal inference, and marketplace analytics—covering A/B test design, randomization and interference concerns, auction and budget effects, metric selection (primary, diagnostic, guardrail), and power/sample-size reasoning.
Meta
Jan 17, 2026, 12:00 AM
Data Scientist
Technical Screen
Analytics & Experimentation
7
0
Loading...
You work on a marketplace with shop ads. A new ranking/recommendation algorithm is proposed to promote shop ads more aggressively, but stakeholders are unclear on what exactly it changes and how to judge whether it is better.
Design an evaluation plan with an online experiment.
What to cover
- Clarify the product change : What questions would you ask to understand what the new algorithm is optimizing/promoting (e.g., different inventory, different bids, different relevance model)?
-
Experiment design
:
- How would you set up an A/B test ?
- What is the correct randomization unit (user vs session), and what are the tradeoffs?
- Any concerns about interference/market-level effects (e.g., auctions, budgets) and how you’d mitigate them.
-
Metrics
:
- Propose primary , diagnostic , and guardrail metrics.
- Include at least: CTR rising but revenue falling (a metric conflict) — how would you interpret and debug it?
-
Power / sample size
:
- Outline how you’d do power analysis (inputs needed, MDE, variance estimation), and what you’d do if the experiment is underpowered.
-
Decision & follow-ups
:
- How you would segment results (e.g., by geography) and decide whether to ship, iterate, or roll back.
- Any offline evaluation you’d do before/alongside the A/B test.
State any assumptions you make (auction type, billing model CPC/CPM, budgets, etc.).