Choosing Causal Inference Methods: Assumptions, Remedies and RCT Power

In a technical interview for an economist internship, the interviewer describes a business case in which the team needs the causal effect of an intervention on an outcome, and asks how you would estimate it. The discussion covers randomized controlled trials (RCTs), propensity score matching (PSM), difference-in-differences (DiD), regression discontinuity (RD) and instrumental variables (IV). The interviewer probes each method's assumptions and intuition and how you would make up for its weaknesses, and a quick, shallow answer can lead to harder probes such as a power calculation for the RCT. The report does not describe the specific case, so this version asks you to work through the methods in a form you can apply to whichever case you are given. ### Clarifying Questions - What is the intervention, and how did units come to receive it: at random, by a rule or score threshold, through a rollout that reached some groups at a known date, or by their own choice? - Is the outcome continuous, such as spend, or binary, such as conversion, and over what window is it measured? - Are outcomes observed for both treated and untreated units before the intervention started? - Which effect does the decision need: the average effect for everyone, for those who were treated, or for those whose treatment a policy change would actually alter? - Can the team still run an experiment, and at what unit (customer, seller, region) can it randomize without units affecting each other? ### Part 1 — Match the method to the case Explain how you would decide which of the five methods fits a case. For each method, state what the assignment of treatment or the available data must look like for it to apply, and which effect it identifies. ```hint Start from assignment Ask first how units ended up treated; each method exploits a different answer to that question. ``` #### What This Part Should Cover - Why randomization is the benchmark, and when it is infeasible or not enough - The assignment pattern or data each observational method requires - The population each method's estimate applies to, and whether that matches the decision ### Part 2 — Assumptions, intuition and remedies For PSM, DiD, RD and IV, state the identifying assumption in plain words, explain the intuition for why the estimate is causal when it holds, describe how you would check it with data, and say what you would do when it is doubtful or fails. Also say what can still go wrong in an RCT. ```hint Testable or not For each assumption, separate what data can check, at least indirectly, from what must be argued from knowledge of how the business works. ``` ```hint Falsification For each method, look for an outcome, period, group or threshold where the true effect must be zero, and ask what the method estimates there. ``` #### What This Part Should Cover - Each method's identifying assumption, stated precisely, with its intuition - A concrete diagnostic or falsification test for each assumption - A remedy, alternative design or sensitivity analysis for when the assumption is in doubt ### Part 3 — Power calculation for the RCT Suppose you recommend an RCT that compares the mean outcome of a treatment group and a control group. Explain how you would choose the sample size: derive the sample size per group, explain each input and where its value comes from, and describe how the answer changes for a binary outcome, an unequal split between the groups, randomizing clusters such as regions instead of individuals, and adjusting for data from before the experiment. ```hint Two error rates Write down the distribution of the test statistic when there is no effect and when the effect equals the smallest one you care about, then require both error rates at once. ``` #### What This Part Should Cover - A correct derivation showing the roles of the significance level, power, outcome variance and minimum detectable effect - How each input is chosen in practice, and how the sample size turns into test duration - The adjustments for binary outcomes, allocation ratio, clustering and variance reduction ### What a Strong Answer Covers - A method choice driven by how treatment was assigned and by the decision the business has to make - Assumptions stated precisely, with intuition and testable implications, rather than a list of method names - Concrete checks and remedies for each method's weak point - A correct power derivation with realistic, justified inputs - Reasoning explained step by step, with the trade-offs between methods made explicit ### Follow-up Questions - In a DiD analysis, the treated and control groups' trends diverge before the intervention. What would you do? - In an RD design, units can nudge their score just above the cutoff. How would you detect this, and how does it affect the estimate? - Your instrument is only weakly related to treatment. What goes wrong, and how would you detect and handle it? - After matching, a key covariate is still imbalanced and many treated units have no comparable controls. What would you do?