Evaluate emoji reactions launch
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
A messaging app plans to introduce an **emoji reaction** feature: users can **long-press a message for 5 seconds** and attach an emoji instead of sending a text reply.
You are the data scientist supporting this launch. Describe how you would evaluate whether this feature is successful, how you would measure its impact, and how you would report results to C-level executives.
Your answer should cover:
1. **Product objective:** What user problem is this feature solving, and what hypotheses would you test?
2. **Success metrics:** Define a metric hierarchy including:
- a primary success metric,
- leading adoption/engagement metrics,
- downstream product metrics,
- guardrail metrics.
3. **Experiment design:** Explain whether you would run an A/B test, staged rollout, or another design. Be explicit about:
- unit of randomization,
- network/interference effects between users in the same conversation,
- required logging/events,
- analysis window.
4. **Risks and tradeoffs:** Discuss possible negative effects such as reactions cannibalizing text replies, accidental long-presses, notification overload, slower messaging UX, or heterogeneous effects across user segments.
5. **Executive reporting:** If you had to summarize the launch to the CEO or other C-level stakeholders, what would you report, how would you frame the business impact, and what recommendation would you make?
Assume the app cares about both **user engagement** and **healthy communication quality**, not just raw feature usage.
Quick Answer: This question evaluates competency in product analytics, experimentation design, metric hierarchy definition, causal inference for social/messaging features, and executive-level communication.