Diagnose post-release conversion regression rigorously
Overview
This question evaluates a Data Scientist's skills in causal inference, experimental design, statistical power and variance-reduction techniques, identification strategies (e.g., DiD and synthetic controls), segmentation and multiple-testing control, guardrail and falsification metrics, and regression-discontinuity reasoning.
Apple
Oct 13, 2025, 9:49 PM
Data Scientist
Technical Screen
Analytics & Experimentation
0
0
Loading...
On July 15, 2025, version 7.3 was fully rolled out. Daily signup conversion fell from 5.4% (July 1–14 baseline) to 4.9% (July 15–21), while spend and traffic mix appear stable. Design a rigorous analysis to determine whether v7.3 caused the −0.5pp drop and what to do next. Specify:
- Identification strategy: e.g., geo‑based holdout (10% traffic on v7.2) with Difference‑in‑Differences, or synthetic control using pre‑period predictors; include formulas and assumptions (parallel trends, SUTVA).
- Guardrail metrics (latency, crash rate, page load) and falsification checks (invariant metrics like bot share).
- CUPED or pre‑period covariate adjustment to reduce variance; compute MDE given N=3,000,000 sessions/day, α=0.05, power=0.8.
- Segmentation plan (device, country, channel cohorts) with multiple‑testing control (e.g., BH‑FDR) and a pre‑registered decision rule for rollback.
- Sensitivity analyses: day‑of‑week seasonality, exposure dosage, novelty effects, and a regression discontinuity at rollout time.
- Concrete outputs: SQL to build daily cohorts, an experiment notebook outline, and the exact criteria you would use to recommend rollback vs. mitigation.