Design metrics and geo A/B for new feature
Overview
This question evaluates skills in experimental design, metric definition and diagnostic analysis for marketplace features, covering geo-clustered A/B testing, hypothesis formulation, power/MDE calculations, instrumentation, contamination handling, privacy considerations and incremental revenue estimation.
Meta
Oct 13, 2025, 9:49 PM
Data Scientist
Onsite
Analytics & Experimentation
0
0
Marketplace Experiment: Verified Seller Badges
Context: You are evaluating a new Marketplace feature, Verified Seller Badges, designed to improve buyer trust and monetization without harming user experience. Propose an end-to-end experiment and analysis plan.
- Mission and Hypotheses
- State a precise mission for the feature.
- Write falsifiable primary and secondary hypotheses (with clear success/fail thresholds).
- Metrics
- Define a single North Star Metric (e.g., weekly GMV per active buyer).
- Provide 4–6 supporting metrics, including at least two counter/guardrail metrics (e.g., fraud reports per 1,000 transactions, session crash rate, ad revenue per session).
- For each metric, explain why it is diagnostic and how to compute it at both user- and geo-level granularity.
- Experiment Design (Geo-Level Clustered A/B)
- Specify the cluster unit (e.g., city/metro) and why.
- Define stratification variables (e.g., active buyers, baseline GMV, seasonality, device mix).
- Describe the matching strategy, number of clusters per arm, traffic ramp plan, duration.
- Explain how you will handle contamination, spillovers, and staggered rollouts.
- Sample Size and Power
- Show how you estimate the minimum detectable effect (MDE) for the North Star Metric, including variance assumptions and design effects due to clustering.
- Instrumentation
- List the exact events and attributes needed in logs to compute all metrics and to diagnose the mechanism (e.g., badge impressions, seller profile views, message initiations, purchase confirmations).
- Decision Framework
- Suppose the test reads: +2.0% (p<0.05) on NSM, −0.3% (ns) on sessions/user, and +0.8 bps in fraud reports (p=0.06).
- Explain your launch decision, incorporating engineering cost, staffing, and operational feasibility.
- Provide a back-of-the-envelope incremental revenue estimate assuming $0.50 revenue per incremental purchase and the observed lift.
- External Data
- Name one third-party signal to improve targeting.
- Discuss privacy/compliance considerations and how to validate its incremental value without bias.
Solution
Login required
Show