VP and PM jointly evaluate how a candidate designs and interprets experiments and defines product metrics when running many concurrent tests under tight timelines.

You must run roughly 100 experiments but lack the traffic to let each reach the usual 0.05 significance level. How would you adjust the overall alpha (Type-I error) threshold so the launched features are as impactful as possible? After seeing preliminary results, how would you incorporate an adaptive approach—such as a multi-armed bandit—to update decision thresholds over time? For a new feature, define the primary success metric you would track, explain why it matters, and describe how you would guard against metric swamping or gaming.

Discuss multiple‐testing corrections (Bonferroni, Holm, FDR), sequential testing, power vs. speed trade-offs, and principled metric selection (north-star, guardrails).

Betterhelp Data Scientist interview question: Optimize Experiment Thresholds for Impactful Feature Launches. {"blocks": [{"key": "0ff314e6", "text": "Scenario", "type": "header-two", "depth": 0, "inlineStyleRanges": [], "entityRanges": [], "data": {...