Experimentation Deep Dive: Attribution, Quasi-Experiments and Cross-Platform Tests

Walk the interviewer through a past project in which you ran an experiment or estimated the causal effect of a product change. Expect probes on how the experiment was designed and read, how the project's impact was attributed, how you would estimate an effect when randomization was not possible, how you would run the experiment across platforms, and what product or business decision the work informed. The report says the interviewer deep-dived the candidate's earlier projects and asked about experimentation, attribution, quasi-experiments and causal inference, and cross-platform experiments, with product and business thinking throughout; the exact prompts are not reported. The Parts below turn each named topic into a question to practise with a project of your own. Where your project never faced a situation, answer how you would handle it. ### Clarifying Questions - Should the project be one where I designed the experiment myself, or may I describe one where my main role was the analysis? - Would you like one project in depth, or a different project for each topic? - Are you more interested in the methods, or in how the results changed a product decision? ### Part 1 — The experiment Describe the experiment at the center of your project: the decision it informed, the hypothesis, the randomization unit, the primary and guardrail metrics, how the sample size and duration were chosen, the result, and what the team did with it. What would you design differently today? ```hint Unit and metric first Be exact about the randomization unit and the primary metric's definition; most probes about validity start from those two choices. ``` #### What This Part Should Cover - The business decision and the hypothesis, stated before the design - The design choices (unit, metrics, power, duration) and the reason for each - How the result was read and turned into a decision, and an honest critique of the design ### Part 2 — Attribution After your change launched, a top-line metric rose, while other launches, marketing campaigns and seasonality also affected it. How much of the rise was due to your project, and how would you show it? ```hint The missing world Attribution asks what the metric would have been without your change. Ask what data could stand in for that unobserved path. ``` #### What This Part Should Cover - The difference between a change's incremental effect and a decomposition of the top-line movement - Methods that build a counterfactual for the launched change, and their assumptions - How individual experiment results relate to the total impact reported for a period ### Part 3 — When you cannot randomize Suppose the change could not be A/B tested: it launched to everyone at once, rolled out market by market, or was available only to users who opted in. How would you estimate its causal effect, and which assumptions would you have to defend? #### Clarifying Questions for this Part - Was the rollout staggered by market or date, and was the order chosen for reasons related to the outcome? - Is there pre-launch history of the outcome metric for both the units that got the change and those that did not? ```hint Who did not get it Look for users or markets that did not get the change, or got it later, and ask what must be true for them to stand in for the treated group. ``` #### What This Part Should Cover - The choice of quasi-experimental method for each rollout pattern - The identifying assumption behind each method and how to probe it - How self-selection into an opt-in feature biases a naive comparison, and how to handle it ### Part 4 — Experimenting across platforms The change ships on mobile, desktop and web, and many users use more than one of them. How would you design the experiment so that assignment, exposure and metrics are consistent across platforms? #### Clarifying Questions for this Part - Are users logged in on every platform, or is some usage anonymous? - Can the change go live on every platform at once, or do some platforms need an app release first? ```hint One person, many devices Ask which identifier exists for the same person on every platform, and what happens to a person who is logged out on one of them. ``` #### What This Part Should Cover - The randomization unit and how assignment stays consistent across devices - Exposure, triggering and release timing that differ by platform - Metrics that capture a user's behavior across platforms, including shifts between them ### What a Strong Answer Covers - A project told in a clear structure, with the candidate's own role and decisions distinct from the team's - Experimental design choices that are sound and justified by the decision the experiment served - Attribution treated as a counterfactual question, with methods matched to the available data - Quasi-experimental methods chosen for the rollout pattern, with their assumptions stated and checked - A cross-platform design that avoids inconsistent assignment and misleading per-platform readings - A link from the analysis to a product or business decision, and honest reflection on what went wrong ### Follow-up Questions - Your experiment showed a significant lift, but after launch the top-line metric did not move. What explanations would you investigate? - You estimate a market-by-market rollout with difference-in-differences, but the treated and comparison markets were not trending in parallel before launch. What do you do? - Leadership wants to report the year's impact by adding up the measured lifts of every launched experiment. What is wrong with that, and what would you propose? - A feature exists only on mobile. How do you measure its effect on a user's total engagement, including desktop and web?