Diagnose drop and assess metric change impact
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
The metric "average number of posts per DAU" drops from 3.0 to 2.5 between 2025-08-31 and 2025-09-01. 1) List and prioritize at least 8 concrete hypotheses spanning product, data, seasonality, and traffic-mix causes (e.g., UI change in the composer flow, posting frictions, rate limits, instrumentation errors, bot mitigation, geo mix shift, app version rollout, outage). For each, specify one falsifiable check and the exact slice you would inspect. 2) Design a same-day triage plan to distinguish measurement bugs from true behavior change: what counters, logs, or A/A checks would you run, and what thresholds decide a bug? 3) If a new feature shipped on 2025-09-01 in the composer, propose an A/B test or phased rollout plan to judge if the change is good or bad: define primary metric(s), at least three guardrail metrics (e.g., crash rate, DAU retention D+1, session length), and key segments (country, platform, new vs returning). 4) Specify decision criteria (minimum effect size and statistical thresholds) and the data collection window you would choose to avoid day-of-week bias. 5) Describe how you would validate that country-level impacts are not masking a global average (Simpson’s paradox) and what action you would take if one large country drives the drop.
Quick Answer: This question evaluates a data scientist's competency in diagnostic analytics, instrumentation validation, causal attribution, experimentation design, and statistical reasoning (including detection of Simpson’s paradox) for product metrics.