Analyze Call Drop Rates Pre- and Post-Update Implementation
Engineers shipped a new Google Meet version intended to reduce call drops, but a traditional A/B test was not possible. You need to estimate whether the new version was effective.
Constraints & Assumptions
-
Treat this as a non-experimental causal-inference problem.
-
Assume call-level logs include drop outcome, timestamp, app version, adoption time, device, network, geography, and relevant covariates.
-
Build a counterfactual for what call-drop rate would have been without the release.
-
Account for rollout timing, seasonality, and user or device mix changes.
Clarifying Questions to Ask
-
Was the rollout gradual by market, device, enterprise policy, or app-store wave?
-
Are users self-selecting into the update?
-
Were other call-quality or infrastructure changes shipped at the same time?
-
What is the exact call-drop definition and denominator?
Part 1 - Analytical Methods
What methods would you use to determine whether the new version is effective without a traditional experiment?
What This Part Should Cover
-
Pre-post interrupted time series, synthetic control, Bayesian structural time series, difference-in-differences, event studies, holdout markets, and regression discontinuity in time.
-
When each method is appropriate and its assumptions.
Part 2 - Data and Metrics
What data and metrics would you use?
What This Part Should Cover
-
Drop rate, reconnect rate, call duration, latency, device/network covariates, app version, rollout timing, meeting size, region, and enterprise account.
-
Guardrails such as crashes, CPU/battery impact, latency, and user complaints.
Part 3 - Validate the Result
How would you validate that the observed change is caused by the update?
What This Part Should Cover
-
Pre-trends, placebo dates, unaffected segments, robustness checks, covariate balance, and sensitivity analysis.
-
Handling staggered adoption and selection bias.
What a Strong Answer Covers
A strong answer builds a credible counterfactual, uses rollout variation when available, checks assumptions carefully, and reports uncertainty rather than relying on a naive pre-post comparison.
Follow-up Questions
-
What if early adopters have better devices and lower baseline drop rates?
-
How would you handle a concurrent infrastructure change?
-
Which analysis would you present to leadership and why?