Design an interference-robust A/B test for monetization
Overview
This question evaluates experimental design and causal inference skills within Analytics & Experimentation, emphasizing interference mitigation, clustering and randomization choices, eligibility and exposure rules, metric hierarchy and guardrails, power and duration planning, variance-reduction techniques, and cluster-robust inference in a two-sided marketplace. It is commonly asked to test a data scientist's practical-application ability to balance monetization and growth trade-offs through precise statistical planning, pre-registered stopping rules, SRM detection, MDE and rollout decision thresholds rather than purely conceptual understanding.
A/B Test Design: New Tipping UI on Creator Posts
Context: You are launching a new tipping UI on creator (PGC/OGC) posts to increase creator monetization. The test must be robust to cross-user interference and two‑sided marketplace (viewer–creator) supply–demand dynamics, while protecting growth and traffic.
Design an experiment that addresses the following:
-
Randomization unit and clustering
- Choose a randomization unit and clustering strategy (e.g., creator-level, ego-network, geo-level) to mitigate cross-user and cross-post spillovers and marketplace substitution.
-
Eligibility and exposure rules
- Define eligibility and exposure rules that prevent treatment contamination across regions and time zones (e.g., US and Asia), including assignment persistence and start/stop timing.
-
Metrics and guardrails
- Provide a primary metric hierarchy (e.g., payer conversion per DAU, ARPPU, creator revenue share) and guardrails (e.g., retention, session length, abuse reports, ad revenue cannibalization). Include precise metric formulas.
-
Power, duration, ramp, and SRM
- Power and duration that cover at least one weekly cycle; a ramp plan; sample ratio mismatch (SRM) detection; and pre-registered stopping rules.
-
Variance reduction and inference
- Variance reduction (e.g., CUPED covariates such as pre-experiment spend and creator popularity) and the exact cluster-robust inference approach.
-
Decision thresholds and rollout
- Decision thresholds and how you would roll out if treatment improves monetization but slightly hurts growth. Be specific about exact formulas and the minimal detectable effect (MDE) you target.
Solution
Login required