Should you roll out if NSM decreases?

Q: Should you roll out if NSM decreases?

This question evaluates proficiency in experimental analysis, metric prioritization, and interpreting trade-offs between a north star metric and secondary signals within A/B testing, testing core data science competencies in the Analytics & Experimentation domain.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Scenario

You ran an experiment. The north star metric (NSM) is profit per order.

Observed results

Average order volume increased in treatment vs control.
Profit per order decreased (statistically and/or practically meaningfully).

Task

Should you roll out the change? Explain your decision process.

Requirements

In your answer, cover:

Why optimizing the NSM matters vs secondary metrics.
What additional checks you would run (segment analysis, guardrails, novelty effects, heterogeneous treatment effects).
When (if ever) you would still consider launching (e.g., if total profit increases, long-term effects, strategic goals).
A clear final recommendation and next steps.

Should you roll out if NSM decreases?

Quick Overview

Scenario

Observed results

Task

Requirements

Solution

Comments (0)