Launch Decisions And Metric Tradeoffs

What's being tested
Ability to translate product tradeoffs into measurable decision rules: pick primary metric(s), set guardrails, quantify expected value and risk, and design experiments/launch plans that detect real business impact. Interviewers expect clarity on statistical power, heterogeneity, ramping, and long-term vs short-term metric conflicts.

Core knowledge

Worked example
Example question: "Should we launch a feature that increases time on site by 8% but reduces 28-day retention by 1.5%?" First, define the objective (e.g., long-term MAU or revenue) and list guardrails (retention, DAU, ad metrics). Translate the percent changes into expected lifetime value per user using historical retention-to-LTV mapping. Estimate whether the short-term engagement uplift plus monetization offsets the retention loss, account for uncertainty via confidence intervals, and propose an A/B with enough power and a staged rollout to watch downstream metrics before full launch.

A common pitfall
The tempting fix is to optimize the largest short-term signal (e.g., engagement minutes) and call it a win. That often ignores lagged harms (churn, ad yield) and novelty spikes. Another common error is underpowering tests or stopping early on a "positive" p-value without pre-specifying decision rules, leading to wrong launch choices and downstream revenue surprises.

Further reading