Investigate Traffic Distribution Impact on Retention Decrease
Quick Overview
Evaluates A/B test diagnostics for a retention decrease after a button-color change. Strong answers check allocation, balance, logging, maturation, statistical robustness, segments, and decision criteria before blaming treatment.
Investigate Traffic Distribution Impact on Retention Decrease
Company: TikTok
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
An A/B test changed a button’s color from green to red; retention dropped. Stakeholders suspect traffic allocation problems.
##### Question
How would you investigate whether the traffic distribution between control and treatment caused the retention decrease? Which diagnostics, balance checks, or statistical tests would you run before concluding the new color hurts retention?
##### Hints
Discuss randomization sanity checks, covariate balance, sequential testing bias, and segment-level retention comparisons.
Overview: Evaluates A/B test diagnostics for a retention decrease after a button-color change. Strong answers check allocation, balance, logging, maturation, statistical robustness, segments, and decision criteria before blaming treatment.
A/B Test Diagnostics: Traffic Distribution and Retention Drop
An A/B test changed a button color from green in control to red in treatment. The primary metric, such as Day-7 retention, decreased in treatment. Stakeholders suspect the drop could be due to traffic allocation or experiment setup issues rather than the color itself.
Outline a step-by-step plan to investigate before concluding that the red button harms retention.
Constraints & Assumptions
Assume nominal 50/50 user-level randomization unless diagnostics show otherwise.
Retention should be measured only for cohorts with enough maturation time.
Sufficient sample size exists for standard asymptotic tests.
Separate assignment problems, exposure problems, instrumentation problems, and true product effects.
Clarifying Questions to Ask Guidance
Is assignment sticky at the user level across devices and sessions?
When was the test launched, and were there ramp changes or incidents?
Which users are eligible, and did eligibility differ between variants?
How is Day-7 retention defined and matured?
Part 1 - Randomization and Allocation Checks
Verify whether traffic distribution problems exist.
What This Part Should Cover Guidance
Run sample ratio mismatch tests against the planned allocation.
Check assignment stickiness, duplicate users, cross-device identity, and eligibility filters.
Compare enrollment over time by variant to detect ramp, outage, or logging issues.
Confirm treatment and control saw the intended button experience.
Part 2 - Balance, Instrumentation, and Maturation
Check whether the variants are comparable and metrics are measured correctly.
What This Part Should Cover Guidance
Compare pre-treatment covariates such as geography, platform, app version, traffic source, tenure, and prior activity.
Verify logging parity, event definitions, and denominator consistency.
Ensure retention cohorts have fully matured and are not affected by censoring.
Check bots, employees, repeated exposures, and missing data.
Part 3 - Statistical and Segment Analysis
Assess whether the retention drop is robust after setup checks.
What This Part Should Cover Guidance
Estimate treatment effect with confidence intervals and appropriate tests for proportions.
Segment by platform, geography, app version, tenure, and exposure intensity without over-interpreting noisy slices.
Examine guardrails and leading metrics such as clicks, errors, latency, session starts, and downstream actions.
Consider novelty effects, interference, and multiple testing.
Part 4 - Decision
Explain how you would decide whether to ship, iterate, or rerun the test.
What This Part Should Cover Guidance
If allocation or logging is broken, rerun or repair before making a launch decision.
If diagnostics pass and the negative effect is robust, recommend rollback or iteration.
If results are mixed, weigh retention against user experience and business goals using pre-defined criteria.
Communicate uncertainty and next steps clearly to stakeholders.
Follow-up Questions Guidance
How would you detect sample ratio mismatch statistically?
What would you do if the retention drop appears only on one app version?
How would the plan change if assignment was session-level instead of user-level?