How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at TikTok.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at TikTok during technical interviews.

Investigate Traffic Distribution Impact on Retention Decrease

Quick Overview

This question evaluates a data scientist's competency in A/B test diagnostics, causal inference, and experiment validity checks within the Analytics & Experimentation domain, covering concepts such as sample ratio mismatch (SRM), covariate and exposure balance, sequential testing bias, and segment-level heterogeneity in retention.

A/B Test Diagnostics: Did Traffic Distribution Cause the Retention Drop?

Context

An A/B test changed a button's color from green (control) to red (treatment). The primary metric (e.g., Day-7 user retention) decreased in the treatment. Stakeholders suspect the retention drop could be due to traffic allocation issues rather than the color itself.

Assume:

User-level randomization with a nominal 50/50 split.
Retention is measured on enrolled users with sufficient maturation time (e.g., D7 retention on cohorts enrolled ≥7 days ago).
Sufficient sample size for standard asymptotic tests.

Question

Outline a step-by-step plan to investigate whether traffic distribution problems caused the retention decrease. What diagnostics, balance checks, and statistical tests would you run before concluding the new color harms retention? Discuss:

Randomization sanity checks and sample ratio mismatch (SRM).
Covariate balance and eligibility/exposure balance.
Sequential testing/peeking bias and timing effects.
Segment-level retention comparisons and heterogeneity.

Provide concrete tests, decision thresholds, and how you would interpret outcomes.

Quick Overview

Context

Assume:

User-level randomization with a nominal 50/50 split.

Retention is measured on enrolled users with sufficient maturation time (e.g., D7 retention on cohorts enrolled ≥7 days ago).

Sufficient sample size for standard asymptotic tests.

Question

Randomization sanity checks and sample ratio mismatch (SRM).

Covariate balance and eligibility/exposure balance.

Sequential testing/peeking bias and timing effects.

Segment-level retention comparisons and heterogeneity.

Provide concrete tests, decision thresholds, and how you would interpret outcomes.

Investigate Traffic Distribution Impact on Retention Decrease

Quick Overview

A/B Test Diagnostics: Did Traffic Distribution Cause the Retention Drop?

Context

Question

Solution

Submit Your Answer to Earn 20XP

Investigate Traffic Distribution Impact on Retention Decrease

Quick Overview

A/B Test Diagnostics: Did Traffic Distribution Cause the Retention Drop?

Context

Question

Solution

Submit Your Answer to Earn 20XP