Design an A/B test for pinned-unread feature

Q: Design an A/B test for pinned-unread feature

This question evaluates proficiency in experiment design and causal inference for product analytics, including choosing the randomization unit, defining eligibility and exposure, selecting primary and guardrail metrics, performing power/MDE and sample-size analysis, and addressing identity cross-contamination and network effects.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

Experiment Design: Evaluating a Pinned-Unread Chat Feature

Context

You are evaluating a new messaging feature that pins chats with unread messages to the top of the inbox to help users notice and respond sooner. Design a rigorous A/B test plan that addresses the following:

Tasks

(a) Define the unit of randomization and assignment (user_id vs person_id, given some people have multiple user_ids), and how you will prevent cross-contamination across identities/devices.

(b) Specify eligibility criteria and pre-exposure rules.

(c) Define primary and secondary success metrics (e.g., reply rate within 24h, messages read per day, time-to-first-response, net unread count change), and guardrail metrics (e.g., crash rate, client CPU/memory, API latency).

(d) Describe your power/MDE and sample size approach, accounting for heavy-tailed engagement.

(e) Propose test duration and seasonality controls.

(f) Address novelty/learning effects and propose a ramp plan.

(g) Describe how you will handle network effects (e.g., group chats) and SUTVA risks.

(h) Explain bucketing for users who frequently switch accounts.

(i) List instrumentation and data quality checks (e.g., duplication, clock skew, delayed reads).

(j) Provide a pre-registered analysis plan, including variance reduction (e.g., CUPED) and how you will interpret conflicting metric movements.

Design an A/B test for pinned-unread feature

Quick Overview

Design an A/B test for pinned-unread feature

Experiment Design: Evaluating a Pinned-Unread Chat Feature

Context

Tasks

Write your answer