Analyze Change in App Metrics and Feature Impact
Overview
This question evaluates a data scientist's competency in product analytics, experimentation design, causal inference, metric selection, hypothesis framing across the AARRR funnel, and quantitative impact sizing within the Analytics & Experimentation domain.
Meta
Aug 4, 2025, 10:55 AM
Data Scientist
Onsite
Analytics & Experimentation
0
0
Scenario
A consumer app has either launched a new feature or observed a sudden change in a key metric. You are asked to investigate and drive a decision.
Tasks
- Frame the problem and outline how you would understand what changed, why it changed, and why the business should care.
-
Metrics
- Propose north-star, secondary, and guard-rail metrics.
- Specify the granularity you would track and why.
-
Sizing
- Size the magnitude of the impact or opportunity. State assumptions and show calculations.
-
Hypothesis
- Formulate hypotheses along the user journey (AARRR: Acquisition, Activation, Retention, Revenue, Referral).
- Specify what data and analyses you would use to validate them.
-
Experiment design
- Design an A/B test: randomization unit, key metrics, network-effect concerns, sanity checks, sample size calculation, and launch decision criteria.
-
If an A/B test is infeasible
- Outline suitable quasi-experimental approaches and how you would validate assumptions.
Hint: Think top-down: frame problem → pick metrics → state hypothesis → design experiment or analysis → weigh trade-offs.
Solution
Login required
Show