Diagnose Traffic Allocation in A/B Test Results

Quick Overview

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Diagnose Traffic Allocation in A/B Test Results states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Diagnose Traffic Allocation in A/B Test Results

Company: TikTok

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Technical Screen

##### Scenario Online experimentation and causal analysis for a consumer app. ##### Question An A/B test changed a call-to-action button from green (control) to red (treatment) and retention dropped. Describe the diagnostics you would run to decide whether uneven traffic allocation or other experiment-quality issues drove the result. You are asked to measure the causal impact of receiving negative reviews on a merchant’s coupon repurchase rate. Outline the data you need and the methodology you would use to obtain an unbiased estimate. ##### Hints Think about sample-ratio mismatch checks, covariate balance, time windows, difference-in-differences, matching/propensity scores, or holdout experiments.

Quick Answer: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Diagnose Traffic Allocation in A/B Test Results states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Analytics & Experimentation/TikTok
TikTok logo
TikTok
Aug 4, 2025, 10:55 AM
mediumData ScientistTechnical ScreenAnalytics & Experimentation
5
0

Diagnose Traffic Allocation in A/B Test Results

Scenario

A consumer app ran an A/B test that changed a call-to-action (CTA) button from green (control) to red (treatment). Retention decreased in treatment.

You need to:

  1. Diagnose whether uneven traffic allocation or experiment-quality issues could explain the observed drop in retention.
  2. Separately, estimate the causal impact of receiving negative reviews on a merchant's coupon repurchase rate.

Assume retention is a k-day retention metric (e.g., 7-day retention), and the platform has standard experimentation infrastructure with event logs, feature flags, and user-level randomization. For the reviews question, assume we have time-stamped purchases and reviews at user–merchant level.

Part A — A/B Test Diagnostics for Retention Drop

Describe the diagnostics you would run to determine if uneven traffic allocation or other experiment-quality issues drove the result, and how you would decide whether to trust the result as causal.

Part B — Causal Impact of Negative Reviews on Coupon Repurchase

Outline the data you need and a methodology to obtain an unbiased estimate of the impact of receiving negative reviews on a merchant’s coupon repurchase rate.

Constraints & Assumptions

  • Preserve the scope, facts, inputs, and requested outputs from the prompt above.
  • If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
  • Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask Guidance

  • Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
  • State assumptions about instrumentation, randomization, sample size, and data quality.
  • Separate descriptive analysis from causal claims.

What a Strong Answer Covers Guidance

  • A metric framework with primary, guardrail, and diagnostic metrics.
  • A credible analysis or experiment design with clear assumptions and bias checks.
  • SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
  • An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions Guidance

  • What sanity checks would you run before trusting the result?
  • How would you handle novelty effects, seasonality, or selection bias?
  • What decision would you make if metrics disagree?
Loading comments...