Launch or Hold a Share-Feature Change When Its A/B Test Reports 20% Usage
Company: Meta
Role: Product Manager
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Other
Your team has changed how the "Share" feature works and is evaluating the change in an A/B test. Reviewing the results, you see that the feature's usage stands at 20%. Would you still launch the change? Explain how you would reach the decision.
```hint Anchor the number
Before deciding, pin down which users the 20% is measured over and what the same figure looks like in the control group.
```
### Clarifying Questions
- Is 20% the share of treatment users who used Share, a change relative to the control group, or the share of the treatment group that ever reached the new flow?
- What was the change meant to improve: how many shares happen, how easy sharing is, or what recipients do with what is shared?
- Which primary metric and guardrails were agreed before the test started, and how long has it been running?
- Which surface is the Share feature on, and which kinds of content does the change cover?
### What a Strong Answer Covers
- An explicit reading of the 20% figure and a comparison with the control group
- The change's intended goal, tied to a primary metric fixed before the readout
- Checks on whether the result can be trusted: sample size and significance, duration and novelty, instrumentation, and spillover between users
- Effects beyond the feature itself, on senders, recipients and overall engagement
- A clear ship, iterate or roll-back rule, and what evidence would change it
### Follow-up Questions
- If share volume fell but recipients opened shared content more often, would you launch? Why or why not?
- How would you redesign the test if shares from treatment users mostly reach people in the control group?
- The result is significant after one week, but you suspect a novelty effect. What do you do?
Overview: An experimentation case about a changed Share feature whose A/B test readout shows 20% usage, asking whether you would still launch it. It tests how you interpret an ambiguous metric, compare it against control, check whether the result can be trusted, and set a clear ship, iterate or roll-back decision rule.