Diagnose a 10% DAU drop
Company: Yahoo
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
Yahoo Mail’s DAU on 2025-09-01 is down 10% vs the 7-day baseline (2025-08-25..2025-08-31). You are the on-call product analyst. Outline an end-to-end plan to: (1) verify the drop (metric definition, log health, bot filters, experiment/treatment exclusions, platform/app-version splits); (2) isolate internal vs external causes using a metric tree (DAU = new + returning; returning = retained + resurrected) and segmentations (geo, device, app version, notification eligibility, spam folder rate, inbox load latency, login failures, SMTP delivery latency, outages, release rollouts, paywall/ads); (3) run quick SQL/Python checks to find the top three contributing segments by absolute DAU loss and estimate their contribution; (4) propose a causal test or quasi-experiment (e.g., rollback A/B, holdout by app version, diff-in-diff with synthetic controls using unaffected geos) to validate the suspected cause; (5) recommend immediate mitigations and a monitoring plan with alert thresholds, guardrail metrics (open rate, send success, latency), and a rollback criterion. Be specific about data you’d pull, success metrics, and the order of operations within the first 2 hours.
Overview: This question evaluates a data scientist's competency in diagnosing a sudden DAU (daily active users) drop, testing skills in metric validation, data quality and ETL checks, segmentation and attribution analysis, causal inference, and monitoring design.