Design an A/B test for a Celebrate reaction
Overview
This question evaluates a data scientist's competency in experimental design for friend-based social feeds, including defining business-aligned success metrics, handling network interference, powering tests, planning heterogeneity analyses, and specifying diagnostics and follow-up experiments.
Experiment Design: Adding a "Celebrate" Reaction to WeChat Moments
Context
WeChat Moments is a friend-based social feed with posting, viewing, and reactions. The product team is considering adding a new "Celebrate" reaction alongside existing reactions. You are asked to design an experiment that credibly measures business impact while handling network interference.
Assume you can log impressions, reactions, hides/mutes/reports, friend-graph edges, notifications, and sessions. You can randomize at user, post, or user→post edge level and can also randomize by clusters (e.g., geos or graph partitions) or by time (switchback).
Tasks
- Define success metrics that credibly tie to business outcomes (e.g., 7/30/90-day retention, posting rate) and guardrails (e.g., session length, hide/mute/report rates, feed quality).
- Choose the randomization unit (user, post, or user→post edge) and defend it under network interference; propose a cluster or switchback design to mitigate spillovers.
- Power the test (state effect size assumptions) and specify a staged rollout with sequential monitoring that controls Type I error.
- Plan heterogeneity analysis (e.g., by country, age, new vs. existing users).
- Detect novelty and saturation effects; propose ramp, cooldown, and difference-in-differences diagnostics.
- Define a follow-up experiment to ensure the feature doesn’t incentivize low-quality posting; include pre-registered decision rules for ship/iterate/rollback.
Solution
Login required