How would you evaluate emoji reactions launch?

You work on a Messenger-like chat app (not Meta). The product team plans to ship a new feature: Emoji Reactions (a user can long-press a message for 5 seconds to add a reaction).

As the Data Scientist supporting the launch, answer the following:

1) What are your goals and hypotheses?

• What user problems could this solve (or create)?
• What do you expect to change in user behavior?

2) What metrics would you use to evaluate success?

Define a metric framework that includes:

• Primary success metric(s) (north-star / decision metric)
• Diagnostic metrics (to explain why the primary metric moved)
• Guardrail metrics (to detect harm)

Be explicit about:

• Metric definitions (numerators/denominators)
• Unit of analysis (user, conversation, message)
• Time windows (e.g., D1/D7) and any timezone assumptions

3) How would you measure impact?

Describe an evaluation plan, such as an A/B test or quasi-experiment:

• Experiment design (randomization unit, eligibility, exposure definition)
• Handling network effects (messaging involves multiple users)
• Power/MDE considerations (what you need to estimate and why)
• Common pitfalls (selection bias, logging issues, novelty effects)

4) What instrumentation / logging is required?

List key events and properties you would need to reliably compute the metrics.

5) How would you communicate results to C-level?

Provide a concise executive readout structure:

• What decision you recommend (ship/iterate/rollback)
• Key results with uncertainty
• Risks, tradeoffs, and next steps

Assume you can run experiments and have event logs, but you must propose what to log and how to define success.