Metrics to Request When an Instagram Test Lifts Stories but Cuts Overall Usage
Company: Meta
Role: Product Manager
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Other
An A/B test on Instagram has come back with two results that pull in opposite directions. In the treatment group, usage of Instagram Stories went up by 10%, but overall usage of Instagram went down by 10%. The person who ran the test asks you, as the product manager, what you need from them before you weigh the gain against the loss. Which metrics would you ask for, and how would each one feed into the decision about the change?
```hint Break the totals apart
Before deciding which number matters more, ask for cuts that show where the lost usage went, which users lost it, and whether the change touched people posting Stories as well as people watching them.
```
### Clarifying Questions
- What does "usage" mean in each result: time spent, sessions, daily active users or content views, and is the same measure used for Stories and for the app as a whole?
- What did the treatment change about Stories (placement, ranking, posting tools or ads), and what was the test meant to achieve?
- How long did the test run, how large were the groups, and are both changes statistically significant?
- Was randomization done per user, and could people in the two groups see each other's Stories and posts?
### What a Strong Answer Covers
- Agreement on what usage means and what the test was for before choosing metrics.
- Checks that the result is valid before trusting either number, including the ways the two groups could influence each other.
- A breakdown of the overall loss by surface, by user segment and between creators and viewers.
- Value metrics beyond time spent, covering long-term health, the content ecosystem and the business.
- A clear decision rule that links the metrics to shipping, iterating or rolling back.
### Follow-up Questions
- Revenue per user is unchanged in the treatment group despite the usage drop. Does that change your recommendation?
- The decline turns out to be concentrated in Reels viewing. What would you ask the Stories team to change?
- How would you set up a longer-running holdout to tell whether the Stories gain lasts?
Overview: A product manager metrics question about an Instagram A/B test in which Stories usage rose 10% while overall app usage fell 10%. It tests which metrics a PM would request to judge test validity, where the lost usage came from, user value and revenue before deciding whether to ship the change.