Experiment on increasing order notifications

Quick Overview

This question evaluates a data scientist's skills in experimental design, causal inference, metric definition and aggregation, power/MDE calculation, and cross‑channel attribution for order‑related messaging campaigns.

Experiment on increasing order notifications

Company: DoorDash

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

Context: Notifications team wants to increase order-related push notifications to drive more sessions and conversions. Tasks: 1) Propose an experiment design to measure incremental impact of higher notification frequency and/or smarter timing. Specify randomization unit (user), exposure caps, time-of-day stratification, and controls for cross-channel effects (email/SMS). Consider multi-armed or bandit variants and explain when you would or would not use them. 2) Define primary success metrics (incremental orders, revenue, profit), and guardrails (opt-out rate, uninstall rate, complaint rate, session quality). Include per-user and per-notification level metrics and the aggregation strategy. 3) Measure long-term effects: design a holdout or staggered-rollout with a 4–8 week follow-up, addressing novelty effects, fatigue, and decay. Describe how you will estimate persistent lift (e.g., switchback tests, long-lived holdouts, synthetic controls). 4) Handle interference and repeated exposure: propose analysis that accounts for saturation and diminishing returns (e.g., dose-response curves, exposure-weighted treatment, instrumental variables via randomized send/no-send at trigger level). 5) Define power/MDE assumptions, stopping rules, and a decision framework that balances short-term lift vs long-term retention risk. Include a concrete rollback criterion.

Quick Answer: This question evaluates a data scientist's skills in experimental design, causal inference, metric definition and aggregation, power/MDE calculation, and cross‑channel attribution for order‑related messaging campaigns.

|Home/Analytics & Experimentation/DoorDash
DoorDash logo
DoorDash
Oct 13, 2025, 9:49 PM
hardData ScientistOnsiteAnalytics & Experimentation
22
0

Experiment Design: Increasing Order‑Related Push Notifications

Context

You are asked to design, measure, and make decisions about increasing order‑related push notifications in a consumer mobile app to drive more sessions and conversions. Assume a large active user base across multiple time zones and multiple messaging channels (push, email, SMS). Order‑related pushes include reminders, cart/checkout nudges, reorder prompts, and relevant deal notifications tied to ordering behavior.

Tasks

  1. Experiment design:
    • Propose an RCT to measure the incremental impact of higher notification frequency and/or smarter timing.
    • Specify the randomization unit, exposure caps, time-of-day stratification, and controls for cross‑channel effects (email/SMS).
    • Consider multi‑armed variants and when (not) to use bandits.
  2. Metrics:
    • Define primary success metrics (incremental orders, revenue, profit) and guardrails (opt‑out rate, uninstall rate, complaint rate, session quality).
    • Include per‑user and per‑notification level metrics and describe aggregation.
  3. Long‑term effects:
    • Design a holdout or staggered rollout with a 4–8 week follow‑up.
    • Address novelty effects, fatigue, and decay, and describe how to estimate persistent lift (e.g., switchbacks, long‑lived holdouts, synthetic controls).
  4. Interference and repeated exposure:
    • Propose analysis to account for saturation and diminishing returns (e.g., dose‑response curves, exposure‑weighted treatment, instrumental variables via randomized send/no‑send at trigger level).
  5. Power/MDE, stopping rules, and decisions:
    • Define power/MDE assumptions, stopping rules, and a decision framework balancing short‑term lift vs long‑term retention risk.
    • Include a concrete rollback criterion.
Loading comments...