Decide launch with asymmetric costs

Quick Overview

This question evaluates a data scientist's ability to integrate statistical inference (p‑values, Type I/II errors, and prior probabilities) with decision analysis and financial impact assessment (NPV, rollout and rollback costs) in the context of A/B testing.

Decide launch with asymmetric costs

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

Using the same test (p=0.10), make a launch decision under asymmetric costs. Assumptions: launching incurs $240k/year maintenance and a $120k rollback cost if reversed; if the true effect is at least −0.15 pp monthly churn, incremental gross profit is $900k/year; otherwise the profit lift is $0. Prior probability that the effect is ≥ −0.15 pp is 30%. Options: (A) full launch now; (B) 10% phased ramp with sequential monitoring for 6 weeks; (C) stop and redesign for a new test in 8 weeks. For each option, compute the 12‑month expected NPV (state any discounting you assume), include Type I/II error implications at α=0.05 with reasonable power assumptions, and recommend one option. Explain how limited engineering capacity for the next quarter would change your choice.

Quick Answer: This question evaluates a data scientist's ability to integrate statistical inference (p‑values, Type I/II errors, and prior probabilities) with decision analysis and financial impact assessment (NPV, rollout and rollback costs) in the context of A/B testing.

|Home/Analytics & Experimentation
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
3
0

Launch Decision Under Asymmetric Costs (Experiment p=0.10)

Context

You ran an A/B test on a churn‑reduction feature. The test's p‑value was 0.10 (suggesting weak evidence of benefit, but not statistically significant at α=0.05). You must decide whether and how to launch, acknowledging asymmetric business costs.

Assume:

  • Success threshold: if the true effect is at least −0.15 percentage points monthly churn (i.e., a churn reduction ≥0.15 pp), the annual incremental gross profit is 900k;otherwise900k; otherwise 0.
  • Ongoing maintenance cost once launched: $240k/year.
  • Rollback cost if you reverse a launched feature: $120k (one‑time).
  • Prior probability that the true effect meets the threshold (≥ −0.15 pp): 30%.
  • Time horizon for NPV: 12 months. State any discounting you assume.

Options

A) Full launch now.

B) 10% phased ramp with sequential monitoring for 6 weeks, then a decision.

C) Stop and redesign for a new test in 8 weeks, then a decision.

Task

For each option:

  1. Compute the 12‑month expected NPV, explicitly stating any discounting assumption and any additional testing/decision timing you assume (e.g., when a confirmatory readout happens, and whether rollback occurs upon a negative readout).
  2. Include Type I/II error implications at α = 0.05 with reasonable power assumptions you choose and justify.
  3. Recommend one option and explain your reasoning.
  4. Briefly explain how limited engineering capacity over the next quarter would change your choice or thresholds.
Loading comments...

Browse More Questions

More Analytics & Experimentation•More Data Scientist•Data Scientist Analytics & Experimentation