Diagnose a sudden metric spike or drop

Quick Overview

This question evaluates a data scientist's competency in anomaly investigation, metric instrumentation validation, causal inference and product analytics, including log-level querying, segmentation, hypothesis generation, and rapid impact estimation.

Diagnose a sudden metric spike or drop

Company: TikTok

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

Your daily Checkout Conversion Rate (CCR) jumped from 3.2% to 4.5% on 2025-06-12 and stayed elevated for 3 days. CCR is defined as unique purchasers / unique sessions with at least one add_to_cart. Provide a concrete, step-by-step investigation plan to: 1) validate the movement is real vs. instrumentation, including exact sanity checks (e.g., event volume balance, missingness by logger version, null spikes, late-arriving data, bot/outlier filters), 2) localize the movement with a minimal set of cuts that can reveal Simpson’s paradox (device, app/web, geo, acquisition channel, experiment arms, release version, traffic vendor, new vs. returning users), 3) enumerate and rank at least five hypotheses (e.g., pricing change, shipping promo, ad mix shift, experiment ramp, search relevance tweak, fraud filter change), with a quick back-of-envelope impact estimate for each and the exact query or metric you would pull to confirm/refute it, 4) distinguish product-causal vs. mix-driven effects by proposing a counterfactual/holdout or synthetic control and the shortest path to compute it with existing logs, 5) decide next actions (rollback, continue, ramp) and guardrails with concrete thresholds (e.g., revenue/user, refund rate, support tickets), and 6) state the artifacts you would produce in the first 60 minutes (plots/tables) and the exact time-series tests or seasonality checks you would run to avoid false alarms.

Quick Answer: This question evaluates a data scientist's competency in anomaly investigation, metric instrumentation validation, causal inference and product analytics, including log-level querying, segmentation, hypothesis generation, and rapid impact estimation.

|Home/Analytics & Experimentation/TikTok
TikTok logo
TikTok
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
9
0

Investigate a 3-Day Jump in Checkout Conversion Rate (CCR)

Context

On 2025-06-12, the daily Checkout Conversion Rate (CCR) increased from 3.2% to 4.5% and stayed elevated for 3 days (through 2025-06-14). CCR is defined as:

CCR = unique purchasers / unique sessions with at least one add_to_cart

Assume we can query event logs and order/payment ledgers with fields like: timestamp, event_name, user_id, session_id, order_id, platform (app/web), device (iOS/Android/Desktop), app_version, logger_version, geo_country, acquisition_channel, experiment_id/variant, traffic_vendor_id, is_new_user, price, currency.

Task

Provide a concrete, step-by-step investigation plan to:

  1. Validate the movement is real vs. instrumentation, including exact sanity checks (event volume balance, missingness by logger version, null spikes, late-arriving data, bot/outlier filters, deduping).
  2. Localize the movement with a minimal set of cuts that can reveal Simpson’s paradox (device, app/web, geo, acquisition channel, experiment arms, release version, traffic vendor, new vs. returning users).
  3. Enumerate and rank at least five hypotheses (e.g., pricing change, shipping promo, ad mix shift, experiment ramp, search relevance tweak, fraud filter change), with a quick back-of-envelope impact estimate for each and the exact query or metric you would pull to confirm/refute it.
  4. Distinguish product-causal vs. mix-driven effects by proposing a counterfactual/holdout or synthetic control and the shortest path to compute it with existing logs.
  5. Decide next actions (rollback, continue, ramp) and guardrails with concrete thresholds (e.g., revenue/user, refund rate, support tickets).
  6. State the artifacts you would produce in the first 60 minutes (plots/tables) and the exact time-series tests or seasonality checks you would run to avoid false alarms.
Loading comments...