You join WhatsApp’s Calling org. PM proposes enabling an adaptive codec for unstable networks to reduce call drops. Design an A/B test that: (a) chooses the randomization unit (caller-level, callee-level, dyad-level, or geo cluster) and justifies it under two-sided exposure/interference; (b) defines primary and guardrail metrics precisely (e.g., call-drop rate = dropped calls / initiated calls excluding employee/test accounts; connection latency p95; complaint rate), and specifies the exposure logic (feature ‘on’ only when both sides are treated vs when caller is treated); (c) outlines a ramp plan and spillover checks; (d) handles non-independence (repeat callers) and seasonality; (e) provides a power/SST back-of-envelope: baseline drop rate 3.2%, target 8% relative reduction, alpha=0.05 (two-sided), power=0.8, 30-day test, average 4 calls/day/caller, caller-level ICC ρ=0.10. Compute the design effect and approximate number of unique callers per arm needed under your chosen unit. Finally, specify the estimand and estimator (e.g., ITT at caller-level with cluster-robust SEs) and how you will diagnose interference (e.g., cross-arm caller–callee edges).

This question evaluates experimental design and statistical reasoning for real-time calling services, focusing on two-sided treatment exposure, interference, metric definition, clustering/repeat-measures handling, and power/sample-size computation.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Design an A/B test for WhatsApp call reliability

A/B Test Design: Adaptive Codec for Unstable Networks (WhatsApp Calling)

Context

You join the Calling organization. A PM proposes enabling an adaptive codec that activates on unstable networks to reduce call drops. The codec is two-sided: it can only run when both participants support it. Your task is to design and size an A/B test that accounts for two-sided exposure and network interference.

Assume we are testing on 1:1 calls (audio and/or video). Unless stated, exclude employees/test accounts and spam/abuse traffic. Assume the baseline drop rate below refers to calls under unstable network conditions (the codec's target population).

Tasks

(a) Choose the randomization unit (caller-level, callee-level, dyad-level, or geo cluster) and justify it under two-sided exposure/interference.

(b) Define primary and guardrail metrics precisely, and specify exposure logic (feature on only when both sides are treated vs when caller alone is treated).

(d) Handle non-independence (repeat callers) and seasonality.

(e) Provide a power/SST back-of-the-envelope. Assume:

Baseline drop rate p0 = 3.2%
Target = 8% relative reduction
Alpha = 0.05 (two-sided)
Power = 0.80
Test length = 30 days
Average 4 calls/day per caller
Caller-level ICC ρ = 0.10 Compute the design effect and the approximate number of unique callers per arm needed under your chosen randomization unit.

Finally, specify the estimand and estimator (e.g., ITT at caller-level with cluster-robust SEs), and how you will diagnose interference (e.g., cross-arm caller–callee edges).

Context

Tasks

(a) Choose the randomization unit (caller-level, callee-level, dyad-level, or geo cluster) and justify it under two-sided exposure/interference.

(b) Define primary and guardrail metrics precisely, and specify exposure logic (feature on only when both sides are treated vs when caller alone is treated).

(d) Handle non-independence (repeat callers) and seasonality.

(e) Provide a power/SST back-of-the-envelope. Assume:

Baseline drop rate p0 = 3.2%

Target = 8% relative reduction

Alpha = 0.05 (two-sided)

Power = 0.80

Test length = 30 days

Average 4 calls/day per caller

Caller-level ICC ρ = 0.10 Compute the design effect and the approximate number of unique callers per arm needed under your chosen randomization unit.

Finally, specify the estimand and estimator (e.g., ITT at caller-level with cluster-robust SEs), and how you will diagnose interference (e.g., cross-arm caller–callee edges).

Design an A/B test for WhatsApp call reliability

Quick Overview

A/B Test Design: Adaptive Codec for Unstable Networks (WhatsApp Calling)

Context

Tasks

Solution

Submit Your Answer to Earn 20XP

Design an A/B test for WhatsApp call reliability

Quick Overview

A/B Test Design: Adaptive Codec for Unstable Networks (WhatsApp Calling)

Context

Tasks

Solution

Submit Your Answer to Earn 20XP