Investigate Pop-up Impact on Partner Referral Conversions
Quick Overview
This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Investigate Pop-up Impact on Partner Referral Conversions states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
Investigate Pop-up Impact on Partner Referral Conversions
Company: DoorDash
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
##### Scenario
Partner-referral channel suddenly shows a sharp fall in first-time purchasers after a new splash pop-up asking visitors to download the mobile app was launched.
##### Question
How would you investigate and confirm that the pop-up caused the drop in conversions from partner links? Assuming online traffic is unchanged, how would you quantify the trade-off between app-download uplift and lost first-purchase conversions? What metrics and experiment design would you propose to decide whether to keep, modify, or remove the pop-up?
##### Hints
Map user funnel, build pre/post or A/B test, measure incremental value of app installs vs. lost orders, control for seasonality and partner mix.
Quick Answer: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Investigate Pop-up Impact on Partner Referral Conversions states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
Investigate Pop-up Impact on Partner Referral Conversions
Partner-Referral Conversions Fell After App Pop-up: Diagnose, Quantify, and Decide
Context
You are an analytics data scientist at a consumer marketplace. A blocking splash pop-up prompting visitors to download the mobile app was launched on mobile web. Soon after, the partner-referral channel shows a sharp drop in first-time purchaser conversions. Overall online traffic volume is unchanged.
Tasks
Causal diagnosis: How would you investigate and confirm the pop-up caused the conversion drop from partner links?
Trade-off quantification: Assuming traffic is unchanged, how would you quantify the trade-off between app-download uplift and lost first-purchase conversions?
Decision framework: Which metrics and experiment design would you propose to decide whether to keep, modify, or remove the pop-up?
Clarifying Questions to Ask Guidance
Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
State assumptions about instrumentation, randomization, sample size, and data quality.
Separate descriptive analysis from causal claims.
What a Strong Answer Covers Guidance
A metric framework with primary, guardrail, and diagnostic metrics.
A credible analysis or experiment design with clear assumptions and bias checks.
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
An actionable recommendation that explains trade-offs and next steps.
Follow-up Questions Guidance
What sanity checks would you run before trusting the result?
How would you handle novelty effects, seasonality, or selection bias?