Diagnose metric drop in Ads Manager

Quick Overview

This question evaluates a Data Scientist's competency in operational analytics, metric integrity, root-cause analysis, and experiment interpretation, covering concepts such as logging latency, sample ratio assessment, and product instrumentation awareness.

Diagnose metric drop in Ads Manager

Company: TikTok

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

Your Ads Manager dashboard shows a 15% relative drop in ad-creation completion rate on 2025-06-10 versus its prior 7-day baseline. The metric is defined as completed_creations / initiated_creations per day at the advertiser level. Describe a step-by-step triage and deep-dive plan to: (1) verify the drop (data quality checks, AA tests, sample ratio mismatch, logging latency); (2) localize the issue (slice by geo, platform, entry point, advertiser tenure/spend tier, funnel step, experiment flags, recent UI/BE releases); (3) determine root cause (product change vs. traffic mix vs. seasonality vs. incidents) using specific analyses you would run; (4) quantify impact and propose immediate mitigations; and (5) define decision criteria for rollback vs. continue. Be specific about queries/cuts you would run and guardrail metrics you would check.

Quick Answer: This question evaluates a Data Scientist's competency in operational analytics, metric integrity, root-cause analysis, and experiment interpretation, covering concepts such as logging latency, sample ratio assessment, and product instrumentation awareness.

|Home/Analytics & Experimentation/TikTok
TikTok logo
TikTok
Oct 13, 2025, 9:49 PM
hardData ScientistOnsiteAnalytics & Experimentation
10
0

Investigate a 15% Drop in Ad-Creation Completion Rate

Context

On 2025-06-10, your Ads Manager dashboard shows a 15% relative decrease in the ad-creation completion rate versus the prior 7-day baseline. The metric is defined daily at the advertiser level as:

  • completion rate = completed_creations / initiated_creations

Assume the KPI is reported for all UI-based ad-creation flows (not API) and that daily processing may have up to a few hours of logging latency.

Task

Describe a step-by-step triage and deep-dive plan to:

  1. Verify the drop (data quality checks, A/A tests, sample ratio mismatch, logging latency).
  2. Localize the issue (slice by geo, platform, entry point, advertiser tenure/spend tier, funnel step, experiment flags, recent UI/BE releases).
  3. Determine root cause (product change vs. traffic mix vs. seasonality vs. incidents) using specific analyses.
  4. Quantify impact and propose immediate mitigations.
  5. Define decision criteria for rollback vs. continue.

Be specific about the queries/cuts you would run and the guardrail metrics you would check.

Loading comments...