Analyze Change in App Metrics and Feature Impact
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
##### Scenario
Product analytics case: a new feature is launched or a key metric suddenly changes on a consumer app.
##### Question
Walk me through how you would understand what changed, why it changed, and why the business should care. How would you size the magnitude of the impact or opportunity? State assumptions and show calculations. Which north-star, secondary, and guard-rail metrics would you track? At what granularity and why? Formulate a hypothesis along the user journey (AARRR). What data and analyses would you use to validate it? Design an A/B test: specify randomization unit, key metrics, network-effect concerns, sanity checks, sample-size calculation, and launch-decision criteria. If an A/B test is infeasible, outline a suitable quasi-experimental approach.
##### Hints
Think top-down: frame problem → pick metrics → state hypothesis → design experiment/analysis → weigh trade-offs.
Quick Answer: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Analyze Change in App Metrics and Feature Impact states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.