Past Project Deep Dive: Defend Experiment Design, Metrics and Selection Bias

Present a data science project you have worked on, end to end: the business background, what you personally contributed, the methods you used, and the results. The interviewer then follows up on whatever the project contains. For a project with an experiment or another causal question, the probes reported for this round are the experiment design, the unit of randomization, how the metrics were chosen, selection bias, and the length of the observation period. The report places this at the start of a technical screen for an ads data science team, ahead of SQL questions and an experimentation case, and notes that statistics was tested through the project and the case rather than through standalone formula questions. Choose a project you can defend at that depth; the parts below separate the presentation from the probing. ### Clarifying Questions - How long should the overview run before the questions start? - Should the project include an experiment or a causal analysis, or is a modeling or reporting project acceptable? - How much business detail can be shared when parts of the project are confidential? ### Part 1 — Present the project Walk through the project: the business background, your own contribution, the method, and the results. ```hint Lead with the decision Before describing any method, decide which business decision the project informed and what changed because of it. ``` ```hint Your part, not the team's Prepare to say exactly which choices were yours, because the follow-ups will test them. ``` #### What This Part Should Cover - The business problem and the decision or metric the project was meant to move - The candidate's own contribution, separated from the team's - The method, with the reason it was chosen over a simpler alternative - Results stated in business terms, with their uncertainty and what happened afterwards ### Part 2 — Defend the design under probing Answer the interviewer's follow-ups on the project's experiment design, randomization unit, metric choice, selection bias, and observation period. ```hint One level up, one level down For the randomization unit, ask what would have gone wrong had you randomized at a coarser unit, and at a finer one. ``` ```hint Who ended up in the analysis Check whether any filter used to pick the analyzed units could itself have been affected by the treatment. ``` #### What This Part Should Cover - The randomization unit justified by where the treatment acts and where it can spill over, and an analysis that respects that unit - Primary, secondary and guardrail metrics, with the reason each was chosen and evidence that they were fixed before the results - Concrete sources of selection bias in this project and how they were prevented or corrected - An observation period justified by weekly cycles, novelty effects, outcome lag and power ### What a Strong Answer Covers - A concise, structured narrative that reaches the result within a few minutes - Clear ownership of decisions, including ones that turned out to be wrong - Every design choice defended against a named alternative, not just described - Candid limitations and what the candidate would do differently - Results tied back to the business decision rather than to test statistics alone ### Follow-up Questions - If you had randomized at a different unit, how would the effect estimate and its standard error have changed? - Treated units that actually engaged with the change did much better than the rest of the treatment group. Why can't you report that comparison as the effect? - If the observation period had been twice as long, which metric would you expect to change, and in which direction? - If the project had no experiment, how did you separate the effect of the change from everything else that happened at the same time?