You work on a Messenger-like chat app (not Meta). The product team plans to ship a new feature: Emoji Reactions (a user can long-press a message for 5 seconds to add a reaction).
As the Data Scientist supporting the launch, answer the following:
1) What are your goals and hypotheses?
-
What user problems could this solve (or create)?
-
What do you expect to change in user behavior?
2) What metrics would you use to evaluate success?
Define a metric framework that includes:
-
Primary success metric(s)
(north-star / decision metric)
-
Diagnostic metrics
(to explain
why
the primary metric moved)
-
Guardrail metrics
(to detect harm)
Be explicit about:
-
Metric definitions (numerators/denominators)
-
Unit of analysis (user, conversation, message)
-
Time windows (e.g., D1/D7) and any timezone assumptions
3) How would you measure impact?
Describe an evaluation plan, such as an A/B test or quasi-experiment:
-
Experiment design (randomization unit, eligibility, exposure definition)
-
Handling network effects (messaging involves multiple users)
-
Power/MDE considerations (what you need to estimate and why)
-
Common pitfalls (selection bias, logging issues, novelty effects)
4) What instrumentation / logging is required?
List key events and properties you would need to reliably compute the metrics.
5) How would you communicate results to C-level?
Provide a concise executive readout structure:
-
What decision you recommend (ship/iterate/rollback)
-
Key results with uncertainty
-
Risks, tradeoffs, and next steps
Assume you can run experiments and have event logs, but you must propose what to log and how to define success.