Investigating a Product Usage Drop with Experiments
You observe that product usage fell by 10 percent in the U.S. and 11 percent in Mexico over the same recent period, such as the last one to two weeks versus a prior baseline. No planned outages were announced. You are tasked with diagnosing causes and proposing an experiment to isolate them.
Constraints & Assumptions
-
First confirm the decline is real and measured consistently.
-
Identify confounders that could affect both markets.
-
Use experiments where feasible, but be explicit when quasi-experimental analysis is more realistic.
-
Report causal lift estimates with uncertainty.
Clarifying Questions to Ask
-
How is usage defined, and did the metric definition change?
-
Did any product releases, app versions, ranking changes, notifications, pricing, or campaigns roll out in both countries?
-
Is the decline concentrated by platform, cohort, channel, or app version?
-
Are there external events, holidays, macro changes, or competitor actions in both markets?
Part 1 - Plausible Confounders
Identify plausible confounders that could explain simultaneous declines.
What This Part Should Cover
-
Include measurement or logging changes, app version rollout, product changes, ranking or feed changes, notification changes, marketing spend, external events, seasonality, competitor activity, and traffic mix.
-
Check whether the timing is aligned across markets.
-
Separate global causes from market-specific causes.
Part 2 - A/B Test Design
Design an A/B test or tests to isolate causal drivers.
What This Part Should Cover
-
Define a main hypothesis, treatment, control, randomization unit, eligibility, exposure, and run duration.
-
Use rollback, feature flag holdout, or targeted experiment if a product change is suspected.
-
Include sample size, power, MDE, and analysis plan.
-
Add guardrails for retention, engagement quality, errors, latency, and support.
Part 3 - Controls and Analysis Variables
Specify variables to control for in setup and analysis.
What This Part Should Cover
-
Include country, platform, app version, tenure, acquisition channel, cohort, seasonality, day of week, device, language, and exposure.
-
Use stratification, covariate adjustment, CUPED, or regression adjustment if appropriate.
-
Check SRM, balance, and pre-trends.
Part 4 - Reporting
Explain how you would report findings and causal lift estimates.
What This Part Should Cover
-
Report estimated lift, confidence intervals, p-values where appropriate, segment results, and guardrail effects.
-
Distinguish confirmed cause, likely cause, and unresolved hypotheses.
-
Recommend next actions and monitoring.
Follow-up Questions
-
What if no randomized holdout exists for the suspected product change?
-
How would you handle the fact that both U.S. and Mexico moved at the same time?
-
What would you report if the experiment fixes usage but hurts quality metrics?