Context: Notifications team wants to increase order-related push notifications to drive more sessions and conversions. Tasks: 1) Propose an experiment design to measure incremental impact of higher notification frequency and/or smarter timing. Specify randomization unit (user), exposure caps, time-of-day stratification, and controls for cross-channel effects (email/SMS). Consider multi-armed or bandit variants and explain when you would or would not use them. 2) Define primary success metrics (incremental orders, revenue, profit), and guardrails (opt-out rate, uninstall rate, complaint rate, session quality). Include per-user and per-notification level metrics and the aggregation strategy. 3) Measure long-term effects: design a holdout or staggered-rollout with a 4–8 week follow-up, addressing novelty effects, fatigue, and decay. Describe how you will estimate persistent lift (e.g., switchback tests, long-lived holdouts, synthetic controls). 4) Handle interference and repeated exposure: propose analysis that accounts for saturation and diminishing returns (e.g., dose-response curves, exposure-weighted treatment, instrumental variables via randomized send/no-send at trigger level). 5) Define power/MDE assumptions, stopping rules, and a decision framework that balances short-term lift vs long-term retention risk. Include a concrete rollback criterion.

This question evaluates a data scientist's skills in experimental design, causal inference, metric definition and aggregation, power/MDE calculation, and cross‑channel attribution for order‑related messaging campaigns.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at DoorDash.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at DoorDash during technical interviews.

Experiment on increasing order notifications | DoorDash Interview Question

Experiment Design: Increasing Order‑Related Push Notifications

Context

You are asked to design, measure, and make decisions about increasing order‑related push notifications in a consumer mobile app to drive more sessions and conversions. Assume a large active user base across multiple time zones and multiple messaging channels (push, email, SMS). Order‑related pushes include reminders, cart/checkout nudges, reorder prompts, and relevant deal notifications tied to ordering behavior.

Tasks

Experiment design:
- Propose an RCT to measure the incremental impact of higher notification frequency and/or smarter timing.
- Specify the randomization unit, exposure caps, time-of-day stratification, and controls for cross‑channel effects (email/SMS).
- Consider multi‑armed variants and when (not) to use bandits.
Metrics:
- Define primary success metrics (incremental orders, revenue, profit) and guardrails (opt‑out rate, uninstall rate, complaint rate, session quality).
- Include per‑user and per‑notification level metrics and describe aggregation.
Long‑term effects:
- Design a holdout or staggered rollout with a 4–8 week follow‑up.
- Address novelty effects, fatigue, and decay, and describe how to estimate persistent lift (e.g., switchbacks, long‑lived holdouts, synthetic controls).
Interference and repeated exposure:
- Propose analysis to account for saturation and diminishing returns (e.g., dose‑response curves, exposure‑weighted treatment, instrumental variables via randomized send/no‑send at trigger level).
Power/MDE, stopping rules, and decisions:
- Define power/MDE assumptions, stopping rules, and a decision framework balancing short‑term lift vs long‑term retention risk.
- Include a concrete rollback criterion.

Context

Tasks

Experiment design:

Propose an RCT to measure the incremental impact of higher notification frequency and/or smarter timing.
Specify the randomization unit, exposure caps, time-of-day stratification, and controls for cross‑channel effects (email/SMS).
Consider multi‑armed variants and when (not) to use bandits.

Metrics:

Define primary success metrics (incremental orders, revenue, profit) and guardrails (opt‑out rate, uninstall rate, complaint rate, session quality).
Include per‑user and per‑notification level metrics and describe aggregation.

Long‑term effects:

Design a holdout or staggered rollout with a 4–8 week follow‑up.
Address novelty effects, fatigue, and decay, and describe how to estimate persistent lift (e.g., switchbacks, long‑lived holdouts, synthetic controls).

Interference and repeated exposure:

Propose analysis to account for saturation and diminishing returns (e.g., dose‑response curves, exposure‑weighted treatment, instrumental variables via randomized send/no‑send at trigger level).

Power/MDE, stopping rules, and decisions:

Define power/MDE assumptions, stopping rules, and a decision framework balancing short‑term lift vs long‑term retention risk.
Include a concrete rollback criterion.

Experiment on increasing order notifications

Quick Overview

Experiment Design: Increasing Order‑Related Push Notifications

Context

Tasks

Solution

Comments (0)

Experiment on increasing order notifications

Quick Overview

Experiment Design: Increasing Order‑Related Push Notifications

Context

Tasks

Solution

Comments (0)