Interview concept

Launch Decisions And Metric Tradeoffs

Asked of: Data Scientist

Last updated

Flowchart showing decision steps for launches: define objective, pick primary metric and guardrails, compute MDE/power, experiment design and ramping, monitor segments/confounders, then decision: launch/iterate/kill.

  1. What's being tested
    Ability to translate product tradeoffs into measurable decision rules: pick primary metric(s), set guardrails, quantify expected value and risk, and design experiments/launch plans that detect real business impact. Interviewers expect clarity on statistical power, heterogeneity, ramping, and long-term vs short-term metric conflicts.

  2. Core knowledge

  • Primary vs guardrail metrics: pick one objective, monitor several safety metrics.
  • Leading (engagement) vs lagging (retention, revenue) metric distinctions and lag windows.
  • Minimum detectable effect (MDE), statistical power, and sample-size calculations.
  • Heterogeneous treatment effects: segment-level uplift and risk stratification.
  • Ramping strategies: canary, gradual percentage rollout, kill criteria, and monitoring.
  • Confounders: novelty effects, instrumentation bias, and correlated metrics.
  • Decision framework: expected value (benefit × probability) and Pareto tradeoff analysis.

  1. Worked example
    Example question: "Should we launch a feature that increases time on site by 8% but reduces 28-day retention by 1.5%?" First, define the objective (e.g., long-term MAU or revenue) and list guardrails (retention, DAU, ad metrics). Translate the percent changes into expected lifetime value per user using historical retention-to-LTV mapping. Estimate whether the short-term engagement uplift plus monetization offsets the retention loss, account for uncertainty via confidence intervals, and propose an A/B with enough power and a staged rollout to watch downstream metrics before full launch.

  2. A common pitfall
    The tempting fix is to optimize the largest short-term signal (e.g., engagement minutes) and call it a win. That often ignores lagged harms (churn, ad yield) and novelty spikes. Another common error is underpowering tests or stopping early on a "positive" p-value without pre-specifying decision rules, leading to wrong launch choices and downstream revenue surprises.

  3. Further reading

  • Kohavi, Longbotham, et al., "Trustworthy Online Controlled Experiments" (Microsoft Research, 2020).
  • Evan Miller, "A/B Testing: A Practical Guide" (blog and calculator) — practical pitfalls and stopping rules.

Related concepts