Validity Threats in A/B Tests: Biases That Break Experiment Conclusions

You are interviewing for a product manager role at Dropbox. The interviewer asks you to name the common biases that can make the result of a product experiment, such as an A/B test, untrustworthy, and to explain how each one distorts the conclusion. ```hint Follow the test from start to finish Problems can creep in when users are assigned to groups, while they experience the change, when outcomes are logged and when someone reads the results. Walking through those stages keeps your list organized and complete. ``` ### Clarifying Questions - Should I focus on online A/B tests in the Dropbox product, or also cover user research and surveys? - Is the unit of randomization an individual user, or could it be a team account whose members share files and folders with each other? - Do you want only problems that bias the estimated effect, or also ones that make chance results look real, such as checking results early? ### What a Strong Answer Covers - An organized set of biases grouped by where they enter: assignment, exposure, measurement, analysis and generalization - For each one, how it distorts the measured effect, with a concrete product example - How to detect each one with checks run before, during and after the test - How to prevent or correct each one in the design or the analysis - A view on which threats matter most for a product built around sharing and team accounts ### Follow-up Questions - A test of a new sharing flow shows a lift, but treated users share folders with users in the control group. How would you redesign the experiment? - The treatment group ended up noticeably larger than control even though the split was meant to be even. What could cause this, and can you still trust the result? - The lift is large in the first week and fades by the fourth. Which number would you report, and why?