Decide launch with asymmetric costs
Overview
This question evaluates a data scientist's ability to integrate statistical inference (p‑values, Type I/II errors, and prior probabilities) with decision analysis and financial impact assessment (NPV, rollout and rollback costs) in the context of A/B testing.
Other
Oct 13, 2025, 9:49 PM
Data Scientist
Technical Screen
Analytics & Experimentation
0
0
Launch Decision Under Asymmetric Costs (Experiment p=0.10)
Context
You ran an A/B test on a churn‑reduction feature. The test's p‑value was 0.10 (suggesting weak evidence of benefit, but not statistically significant at α=0.05). You must decide whether and how to launch, acknowledging asymmetric business costs.
Assume:
- Success threshold: if the true effect is at least −0.15 percentage points monthly churn (i.e., a churn reduction ≥0.15 pp), the annual incremental gross profit is 0.
- Ongoing maintenance cost once launched: $240k/year.
- Rollback cost if you reverse a launched feature: $120k (one‑time).
- Prior probability that the true effect meets the threshold (≥ −0.15 pp): 30%.
- Time horizon for NPV: 12 months. State any discounting you assume.
Options
A) Full launch now.
B) 10% phased ramp with sequential monitoring for 6 weeks, then a decision.
C) Stop and redesign for a new test in 8 weeks, then a decision.
Task
For each option:
- Compute the 12‑month expected NPV, explicitly stating any discounting assumption and any additional testing/decision timing you assume (e.g., when a confirmatory readout happens, and whether rollback occurs upon a negative readout).
- Include Type I/II error implications at α = 0.05 with reasonable power assumptions you choose and justify.
- Recommend one option and explain your reasoning.
- Briefly explain how limited engineering capacity over the next quarter would change your choice or thresholds.
Solution
Login required
Show