A/B Test Design for Modifying a Disliked Social Feature

Scenario

A social-media company plans to modify a disliked Instagram-like feature and must run an A/B test before shipping the change.

Task

Pick a popular consumer app and name a feature you do not like. Then answer the following:

1. What single or composite metric would you track to measure user response? State why it aligns with the product goal.
2. How would you estimate the required sample size and decide how long the test should run? Show assumptions and calculations.
3. Which user segment or geography would you target first and why?
4. What statistical test would you apply to compare control and treatment groups and why is it appropriate?
5. Suppose the overall test is not statistically significant but the treatment shows higher engagement. How would you interpret this outcome?
6. Given the above, would you (a) fully launch the change, (b) rerun or extend the experiment, or (c) dig into a specific subgroup (e.g., younger users)? Justify your choice.

Hints: Discuss metric choice, power analysis, duration, segmentation, test selection, and post-test decisions.