Modify Instagram Feature: Track User Engagement Metric
Overview
This question evaluates a data scientist's skills in experimental design, metric definition, statistical power and sample-size estimation, hypothesis testing, and segmentation interpretation for product experiments.
Yelp
Aug 4, 2025, 10:55 AM
Data Scientist
Technical Screen
Analytics & Experimentation
71
0
A/B Test Design for Modifying a Disliked Social Feature
Scenario
A social-media company plans to modify a disliked Instagram-like feature and must run an A/B test before shipping the change.
Task
Pick a popular consumer app and name a feature you do not like. Then answer the following:
- What single or composite metric would you track to measure user response? State why it aligns with the product goal.
- How would you estimate the required sample size and decide how long the test should run? Show assumptions and calculations.
- Which user segment or geography would you target first and why?
- What statistical test would you apply to compare control and treatment groups and why is it appropriate?
- Suppose the overall test is not statistically significant but the treatment shows higher engagement. How would you interpret this outcome?
- Given the above, would you (a) fully launch the change, (b) rerun or extend the experiment, or (c) dig into a specific subgroup (e.g., younger users)? Justify your choice.
Hints: Discuss metric choice, power analysis, duration, segmentation, test selection, and post-test decisions.
Solution
Login required
Show