Design and analyze email deliverability experiment
Company: Microsoft
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
Design a rigorous experiment to test whether sending from Outlook yields higher deliverability to a specific enterprise domain than sending from Gmail. Provide: hypotheses and estimands; experimental unit and randomization scheme that balances time-of-day and content across providers; a two-stage sequential design with an interim at 500 sends when the baseline failure rate is unknown; instrumentation details (SPF, DKIM, DMARC alignment, return-path domain, seed inboxes across subdomains, per-message signed tokens, and server-side web beacons); primary metric (delivered-to-inbox within 5 minutes of send) and secondary metrics (time-to-first-inbox, spam-folder rate, hard-bounce rate); an analysis plan using either a difference-in-proportions test with continuity correction and multiplicity control or a Bayesian beta-binomial with skeptical priors, including decision thresholds and stopping rules; plans to detect and mitigate confounders (content drift, throttling, OOO bursts, holiday effects) and to assess heterogeneity by recipient subdomain; procedures to validate assumptions and generalize results to future campaigns.
Quick Answer: This question evaluates a data scientist's competency in experimental design, causal inference, sequential testing (frequentist and Bayesian), instrumentation for observability, and applied metrics for email deliverability within the Analytics & Experimentation domain.