How would you evaluate emoji reactions launch?

You work on a **Messenger-like chat app** (not Meta). The product team plans to ship a new feature: **Emoji Reactions** (a user can long-press a message for **5 seconds** to add a reaction). As the Data Scientist supporting the launch, answer the following: ## 1) What are your goals and hypotheses? - What user problems could this solve (or create)? - What do you expect to change in user behavior? ## 2) What metrics would you use to evaluate success? Define a metric framework that includes: - **Primary success metric(s)** (north-star / decision metric) - **Diagnostic metrics** (to explain *why* the primary metric moved) - **Guardrail metrics** (to detect harm) Be explicit about: - Metric definitions (numerators/denominators) - Unit of analysis (user, conversation, message) - Time windows (e.g., D1/D7) and any timezone assumptions ## 3) How would you measure impact? Describe an evaluation plan, such as an A/B test or quasi-experiment: - Experiment design (randomization unit, eligibility, exposure definition) - Handling network effects (messaging involves multiple users) - Power/MDE considerations (what you need to estimate and why) - Common pitfalls (selection bias, logging issues, novelty effects) ## 4) What instrumentation / logging is required? List key events and properties you would need to reliably compute the metrics. ## 5) How would you communicate results to C-level? Provide a concise executive readout structure: - What decision you recommend (ship/iterate/rollback) - Key results with uncertainty - Risks, tradeoffs, and next steps Assume you can run experiments and have event logs, but you must propose what to log and how to define success.