PracHub
QuestionsPremiumCoachesLearningGuidesInterview Prep
|Home/Statistics & Math

Decide launch with p=0.10 at alpha 0.05

Last updated: Mar 29, 2026

Quick Overview

This question evaluates a data scientist's competency in statistical inference and experiment decision-making, including frequentist hypothesis testing, conditional power and sample-size planning, Bayesian updating, and cost-sensitive risk analysis.

  • hard
  • Statistics & Math
  • Data Scientist

Decide launch with p=0.10 at alpha 0.05

Role: Data Scientist

Category: Statistics & Math

Difficulty: hard

Interview Round: Technical Screen

An A/B test was pre-registered with α=0.05 (two-sided), power=80%, and MDE=+2% relative lift in 30‑day subscriber retention. After 4 weeks, the observed lift is +1.4% relative with p=0.10; guardrails show no harm. False-positive cost is estimated to be 3× the false-negative cost. (1) Give a binary launch/no‑launch decision under the pre-registered frequentist plan and justify. (2) Compute or outline the conditional power and additional sample size/duration needed to achieve 80% power at a 1.4% MDE; recommend extend vs stop. (3) Reframe the decision with a simple Bayesian analysis (e.g., Beta‑Binomial or normal approximation) using a weakly informative prior; compare expected loss for launch vs no‑launch under the 3:1 cost ratio. (4) If choosing to ship despite p=0.10, propose a risk‑mitigated rollout (e.g., phased ramp, sequential testing/CUPED, kill switches) and explicit rollback criteria.

Quick Answer: This question evaluates a data scientist's competency in statistical inference and experiment decision-making, including frequentist hypothesis testing, conditional power and sample-size planning, Bayesian updating, and cost-sensitive risk analysis.

Related Interview Questions

  • Commuter Ride-Choice Probability: Committed and Permanent Riders - Lyft (medium)
  • Measure Bird Species Segregation - Google (medium)
  • Compute Commuter Ride Probabilities - Lyft (medium)
  • Compute Parking Clustering Probability and P-Value - Gusto (medium)
  • Interpret and Regularize Regression Models - Instacart (hard)
 logo
Oct 13, 2025, 9:49 PM
Data Scientist
Technical Screen
Statistics & Math
0
0
Loading...

A/B Test Decision: Frequentist, Conditional Power, Bayesian, and Risk-Mitigation

Context

You ran a 50/50 A/B test on 30‑day subscriber retention. The pre‑registered analysis plan specified:

  • Two‑sided α = 0.05
  • Power = 80%
  • MDE = +2% relative lift in 30‑day retention

After 4 weeks:

  • Observed effect: +1.4% relative lift
  • Two‑sided p‑value: 0.10 (guardrails show no harm)
  • Business costs: false positives cost 3× false negatives

Assume equal allocation, large‑sample normal approximations are valid, and metric maturity is properly handled for the observed p‑value.

Tasks

  1. Under the pre‑registered frequentist plan, give a binary launch/no‑launch decision and justify.
  2. Compute or outline the conditional power and the additional sample size/duration needed to achieve 80% power for a 1.4% relative lift; recommend extend vs stop.
  3. Reframe with a simple Bayesian analysis (weakly informative prior) and compare expected loss for launch vs no‑launch under the 3:1 cost ratio.
  4. If shipping despite p = 0.10, propose a risk‑mitigated rollout (phased ramp, sequential testing/CUPED, kill switches) and explicit rollback criteria.

Solution

Show

Submit Your Answer to Earn 20XP

Sign in to leave a comment

Loading comments...

Browse More Questions

More Statistics & Math•More Data Scientist•Data Scientist Statistics & Math
PracHub

Master your tech interviews with 8,000+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.