Measure Whether Facebook Messenger Product Changes Worked: Metrics and Experiment Plan
Company: Meta
Role: Product Manager
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Other
You are a product manager on Facebook Messenger. How would you measure the improvements your team makes to Messenger? Explain which metrics would show whether a change made Messenger better, and how you would attribute any movement in them to the change.
```hint Start from a conversation
Think about what people come to Messenger to do, trace the steps from opening a chat to getting a reply, and ask which of those steps an improvement is meant to change.
```
### Clarifying Questions
- Is there one specific improvement to evaluate, such as a new message type, a group feature or faster sending, or should the answer work for any change the team ships?
- Can the change be tested against a control group, or has it already launched to everyone?
- Which surfaces are in scope: the standalone Messenger app, chats inside the Facebook app, or the web as well?
- Should success mean more conversations, more time spent in Messenger, or better conversations at the same level of use?
### What a Strong Answer Covers
- Messenger's goal and the user behaviors that reflect it, such as sending, reading, replying, group chats and calls
- One clearly defined primary metric that reflects successful conversations rather than raw activity
- Supporting metrics for adoption, depth of engagement and retention, plus guardrails such as unwanted messages, blocks, failed sends and crashes
- An experiment design that accounts for messages flowing between users in test and control
- How the metric set changes for a specific improvement, and how to decide when metrics disagree
### Follow-up Questions
- After a change, messages sent per user rise but the share of messages that get a reply falls. Was the change an improvement?
- How would you judge a change meant to make messaging faster, where people may end up spending less time in the app?
- Messaging also happens in other Meta apps. How would you tell whether a gain in Messenger is new conversation or activity moved from another app?
Overview: A Meta product manager interview question on measuring improvements to Facebook Messenger. It tests setting a goal for a messaging product, choosing a primary metric with supporting and guardrail metrics, and designing experiments where users in test and control message each other.