A/B Test Design: Adaptive Codec for Unstable Networks (WhatsApp Calling)

Context

You join the Calling organization. A PM proposes enabling an adaptive codec that activates on unstable networks to reduce call drops. The codec is two-sided: it can only run when both participants support it. Your task is to design and size an A/B test that accounts for two-sided exposure and network interference.

Assume we are testing on 1:1 calls (audio and/or video). Unless stated, exclude employees/test accounts and spam/abuse traffic. Assume the baseline drop rate below refers to calls under unstable network conditions (the codec's target population).

Tasks

(a) Choose the randomization unit (caller-level, callee-level, dyad-level, or geo cluster) and justify it under two-sided exposure/interference.

(b) Define primary and guardrail metrics precisely, and specify exposure logic (feature on only when both sides are treated vs when caller alone is treated).

(c) Outline a ramp plan and spillover checks.

(d) Handle non-independence (repeat callers) and seasonality.

(e) Provide a power/SST back-of-the-envelope. Assume:

• Baseline drop rate p0 = 3.2%
• Target = 8% relative reduction
• Alpha = 0.05 (two-sided)
• Power = 0.80
• Test length = 30 days
• Average 4 calls/day per caller
• Caller-level ICC ρ = 0.10 Compute the design effect and the approximate number of unique callers per arm needed under your chosen randomization unit.

Finally, specify the estimand and estimator (e.g., ITT at caller-level with cluster-robust SEs), and how you will diagnose interference (e.g., cross-arm caller–callee edges).