Design an A/B test for pinned-unread feature
Overview
This question evaluates proficiency in experiment design and causal inference for product analytics, including choosing the randomization unit, defining eligibility and exposure, selecting primary and guardrail metrics, performing power/MDE and sample-size analysis, and addressing identity cross-contamination and network effects.
Experiment Design: Evaluating a Pinned-Unread Chat Feature
Context
You are evaluating a new messaging feature that pins chats with unread messages to the top of the inbox to help users notice and respond sooner. Design a rigorous A/B test plan that addresses the following:
Tasks
(a) Define the unit of randomization and assignment (user_id vs person_id, given some people have multiple user_ids), and how you will prevent cross-contamination across identities/devices.
(b) Specify eligibility criteria and pre-exposure rules.
(c) Define primary and secondary success metrics (e.g., reply rate within 24h, messages read per day, time-to-first-response, net unread count change), and guardrail metrics (e.g., crash rate, client CPU/memory, API latency).
(d) Describe your power/MDE and sample size approach, accounting for heavy-tailed engagement.
(e) Propose test duration and seasonality controls.
(f) Address novelty/learning effects and propose a ramp plan.
(g) Describe how you will handle network effects (e.g., group chats) and SUTVA risks.
(h) Explain bucketing for users who frequently switch accounts.
(i) List instrumentation and data quality checks (e.g., duplication, clock skew, delayed reads).
(j) Provide a pre-registered analysis plan, including variance reduction (e.g., CUPED) and how you will interpret conflicting metric movements.
Solution
Login required