Design A/B Test to Isolate Product Usage Drop Causes
Quick Overview
Evaluates experimentation strategy for diagnosing a simultaneous product usage drop in the U.S. and Mexico. Strong answers identify confounders, design tests or quasi-experiments, control variables, and report causal lift.
Design A/B Test to Isolate Product Usage Drop Causes
Company: Google
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
Product usage dropped 10 % in the U.S. and 11 % in Mexico.
##### Question
Identify potential confounders, design an A/B test to isolate the cause, specify the variables you would control, and explain how you would report findings to stakeholders.
##### Hints
Segment users, hold out controls, present causal lift estimates.
Quick Answer: Evaluates experimentation strategy for diagnosing a simultaneous product usage drop in the U.S. and Mexico. Strong answers identify confounders, design tests or quasi-experiments, control variables, and report causal lift.
Investigating a Product Usage Drop with Experiments
You observe that product usage fell by 10 percent in the U.S. and 11 percent in Mexico over the same recent period, such as the last one to two weeks versus a prior baseline. No planned outages were announced. You are tasked with diagnosing causes and proposing an experiment to isolate them.
Constraints & Assumptions
First confirm the decline is real and measured consistently.
Identify confounders that could affect both markets.
Use experiments where feasible, but be explicit when quasi-experimental analysis is more realistic.
Report causal lift estimates with uncertainty.
Clarifying Questions to Ask Guidance
How is usage defined, and did the metric definition change?
Did any product releases, app versions, ranking changes, notifications, pricing, or campaigns roll out in both countries?
Is the decline concentrated by platform, cohort, channel, or app version?
Are there external events, holidays, macro changes, or competitor actions in both markets?
Part 1 - Plausible Confounders
Identify plausible confounders that could explain simultaneous declines.
What This Part Should Cover Guidance
Include measurement or logging changes, app version rollout, product changes, ranking or feed changes, notification changes, marketing spend, external events, seasonality, competitor activity, and traffic mix.
Check whether the timing is aligned across markets.
Separate global causes from market-specific causes.
Part 2 - A/B Test Design
Design an A/B test or tests to isolate causal drivers.
What This Part Should Cover Guidance
Define a main hypothesis, treatment, control, randomization unit, eligibility, exposure, and run duration.
Use rollback, feature flag holdout, or targeted experiment if a product change is suspected.
Include sample size, power, MDE, and analysis plan.
Add guardrails for retention, engagement quality, errors, latency, and support.
Part 3 - Controls and Analysis Variables
Specify variables to control for in setup and analysis.
What This Part Should Cover Guidance
Include country, platform, app version, tenure, acquisition channel, cohort, seasonality, day of week, device, language, and exposure.
Use stratification, covariate adjustment, CUPED, or regression adjustment if appropriate.
Check SRM, balance, and pre-trends.
Part 4 - Reporting
Explain how you would report findings and causal lift estimates.
What This Part Should Cover Guidance
Report estimated lift, confidence intervals, p-values where appropriate, segment results, and guardrail effects.
Distinguish confirmed cause, likely cause, and unresolved hypotheses.
Recommend next actions and monitoring.
Follow-up Questions Guidance
What if no randomized holdout exists for the suspected product change?
How would you handle the fact that both U.S. and Mexico moved at the same time?
What would you report if the experiment fixes usage but hurts quality metrics?