Decisions Under Uncertainty and Precommitment
Asked of: Data Scientist
Last updated
-
What it is Decisions under uncertainty are choices you make when outcomes are stochastic or only partially known; you reason with probabilities, risks, and payoffs. Precommitment means fixing your decision policy in advance (e.g., metrics, thresholds, stopping rules) to avoid bias, time-inconsistency, or p-hacking.
-
Why interviewers ask about it At product companies (e.g., large social platforms), data scientists decide launches with noisy, shifting data. They expect you to structure experiments, quantify uncertainty, and precommit to rules so business decisions aren’t swayed by interim fluctuations or post-hoc narratives.
-
Core ideas to know
- Frame choices by expected impact; use uncertainty intervals to communicate risk to PMs and engineers.
- Exploration vs. exploitation: bandits reduce regret when continuously allocating traffic under uncertainty.
- Precommit your hypothesis, primary metric, segments, and guardrails before launch.
- Specify decision thresholds tied to business value, not just p=0.05 or “lift > 0.”
- Avoid optional stopping; use fixed-horizon tests or valid sequential methods with error control.
- Plan sample size via MDE and power; document when you’ll stop, ramp, or roll back.
- Use guardrails (e.g., latency, crash rate, spam) and staged ramps to cap downside risk.
-
A common pitfall Candidates describe A/B testing mechanics but fail to state the precommitted decision rule. They peek daily and propose “stop when significant,” ignoring inflated false positives and regression to the mean. Others swap primary metrics midstream or switch from fixed-horizon to Bayesian post hoc to justify a launch. Interviewers want a crisp, written plan: hypothesis, metric, MDE/power, stopping/analysis method, and the explicit “ship/hold” rule.
-
Further reading
- Bandit Algorithms — Lattimore & Szepesvári (Cambridge, 2020) [free PDF]. Clear treatment of exploration–exploitation, regret, and when bandits beat fixed designs. (tor-lattimore.com)
- Trustworthy Online Controlled Experiments — Kohavi, Tang, Xu (O’Reilly/Cambridge, 2020) Chapter 1 excerpt. Practical checklists on metrics, guardrails, and pre-analysis plans at scale. (experimentguide.com)
- Statistical Challenges in Online Controlled Experiments (arXiv, 2022). Survey of modern A/B testing issues including optional stopping and sequential methods used in industry. (arxiv.org)