How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Other.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Other during technical interviews.

Decide launch with asymmetric costs | Other Interview Question

Quick Overview

This question evaluates a data scientist's ability to integrate statistical inference (p‑values, Type I/II errors, and prior probabilities) with decision analysis and financial impact assessment (NPV, rollout and rollback costs) in the context of A/B testing.

Launch Decision Under Asymmetric Costs (Experiment p=0.10)

Context

You ran an A/B test on a churn‑reduction feature. The test's p‑value was 0.10 (suggesting weak evidence of benefit, but not statistically significant at α=0.05). You must decide whether and how to launch, acknowledging asymmetric business costs.

Assume:

Success threshold: if the true effect is at least −0.15 percentage points monthly churn (i.e., a churn reduction ≥0.15 pp), the annual incremental gross profit is $900k; otherwise$ 0.
Ongoing maintenance cost once launched: $240k/year.
Rollback cost if you reverse a launched feature: $120k (one‑time).
Prior probability that the true effect meets the threshold (≥ −0.15 pp): 30%.
Time horizon for NPV: 12 months. State any discounting you assume.

Options

A) Full launch now.

B) 10% phased ramp with sequential monitoring for 6 weeks, then a decision.

C) Stop and redesign for a new test in 8 weeks, then a decision.

Task

For each option:

Compute the 12‑month expected NPV, explicitly stating any discounting assumption and any additional testing/decision timing you assume (e.g., when a confirmatory readout happens, and whether rollback occurs upon a negative readout).
Include Type I/II error implications at α = 0.05 with reasonable power assumptions you choose and justify.
Recommend one option and explain your reasoning.
Briefly explain how limited engineering capacity over the next quarter would change your choice or thresholds.

Quick Overview

Context

Assume:

Success threshold: if the true effect is at least −0.15 percentage points monthly churn (i.e., a churn reduction ≥0.15 pp), the annual incremental gross profit is

900k; otherwise

Ongoing maintenance cost once launched: $240k/year.

Rollback cost if you reverse a launched feature: $120k (one‑time).

Prior probability that the true effect meets the threshold (≥ −0.15 pp): 30%.

Time horizon for NPV: 12 months. State any discounting you assume.

Task

For each option:

Compute the 12‑month expected NPV, explicitly stating any discounting assumption and any additional testing/decision timing you assume (e.g., when a confirmatory readout happens, and whether rollback occurs upon a negative readout).

Include Type I/II error implications at α = 0.05 with reasonable power assumptions you choose and justify.

Recommend one option and explain your reasoning.

Briefly explain how limited engineering capacity over the next quarter would change your choice or thresholds.

Decide launch with asymmetric costs

Quick Overview

Launch Decision Under Asymmetric Costs (Experiment p=0.10)

Context

Options

Task

Solution

Comments (0)

Decide launch with asymmetric costs

Quick Overview

Launch Decision Under Asymmetric Costs (Experiment p=0.10)

Context

Options

Task

Solution

Comments (0)