Investigate Traffic Distribution Impact on Retention Decrease

Q: Investigate Traffic Distribution Impact on Retention Decrease

This question evaluates a data scientist's competency in A/B test diagnostics, causal inference, and experiment validity checks within the Analytics & Experimentation domain, covering concepts such as sample ratio mismatch (SRM), covariate and exposure balance, sequential testing bias, and segment-level heterogeneity in retention.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at TikTok.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at TikTok during technical interviews.

Question

A/B Test Diagnostics: Did Traffic Distribution Cause the Retention Drop?

Context

An A/B test changed a button's color from green (control) to red (treatment). The primary metric (e.g., Day-7 user retention) decreased in the treatment. Stakeholders suspect the retention drop could be due to traffic allocation issues rather than the color itself.

Assume:

User-level randomization with a nominal 50/50 split.
Retention is measured on enrolled users with sufficient maturation time (e.g., D7 retention on cohorts enrolled ≥7 days ago).
Sufficient sample size for standard asymptotic tests.

Question

Outline a step-by-step plan to investigate whether traffic distribution problems caused the retention decrease. What diagnostics, balance checks, and statistical tests would you run before concluding the new color harms retention? Discuss:

Randomization sanity checks and sample ratio mismatch (SRM).
Covariate balance and eligibility/exposure balance.
Sequential testing/peeking bias and timing effects.
Segment-level retention comparisons and heterogeneity.

Provide concrete tests, decision thresholds, and how you would interpret outcomes.

Investigate Traffic Distribution Impact on Retention Decrease

Quick Overview

A/B Test Diagnostics: Did Traffic Distribution Cause the Retention Drop?

Context

Question

Solution

Comments (0)