Diagnose metric drop in Ads Manager
Overview
This question evaluates a Data Scientist's competency in operational analytics, metric integrity, root-cause analysis, and experiment interpretation, covering concepts such as logging latency, sample ratio assessment, and product instrumentation awareness.
Investigate a 15% Drop in Ad-Creation Completion Rate
Context
On 2025-06-10, your Ads Manager dashboard shows a 15% relative decrease in the ad-creation completion rate versus the prior 7-day baseline. The metric is defined daily at the advertiser level as:
- completion rate = completed_creations / initiated_creations
Assume the KPI is reported for all UI-based ad-creation flows (not API) and that daily processing may have up to a few hours of logging latency.
Task
Describe a step-by-step triage and deep-dive plan to:
- Verify the drop (data quality checks, A/A tests, sample ratio mismatch, logging latency).
- Localize the issue (slice by geo, platform, entry point, advertiser tenure/spend tier, funnel step, experiment flags, recent UI/BE releases).
- Determine root cause (product change vs. traffic mix vs. seasonality vs. incidents) using specific analyses.
- Quantify impact and propose immediate mitigations.
- Define decision criteria for rollback vs. continue.
Be specific about the queries/cuts you would run and the guardrail metrics you would check.
Solution
Login required