## Estimate Incremental Search-Ad Revenue Without an A/B Test A search product assigns 100 users, selected at random from a clearly defined eligible population, to see ads in search results. The other eligible users continue to receive the existing experience without ads. The team cannot use its conventional A/B testing system, but it retains the assignment record and can measure outcomes for both groups. The business wants an estimate of the incremental revenue caused by the ads, not merely the revenue recorded after launch. Explain how you would estimate the causal effect. Define the estimand, identify a credible comparison strategy, describe the data you need, and show how you would translate the estimated effect into incremental revenue. Include the assumptions, diagnostics, uncertainty, and limitations you would communicate to the decision-maker. ### Constraints & Assumptions - The 100 treated users were selected by random assignment from the same launch-eligible population as the untreated users. - Analyze users according to their assigned group for the primary estimate, even if actual ad exposure is imperfect. - Historical data may be available for precision adjustment and diagnostics, but the random assignment—not a parallel-trends assumption—is the primary basis for causal identification. - Revenue can be highly skewed and may be affected by seasonality, user mix, and changes in search activity. ### Clarifying Questions to Ask - What is the intended target population: the 100 treated users, all eligible users, or a future rollout population? - What exactly counts as revenue, over what attribution window, and are we measuring gross ad revenue or a net business outcome? - What was the assignment unit and treatment probability, and can the randomization record be audited? - Did any assigned users fail to receive their experience, cross groups, leave the sample, or become ineligible after assignment? - Could ads displace organic results or affect search volume, retention, or other guardrail metrics? ### What a Strong Answer Covers - Recognizes that the random selection creates a randomized treated-versus-untreated comparison even without conventional experiment infrastructure. - States a causal estimand over a defined outcome window and uses the post-assignment difference in mean revenue per eligible user as the primary intent-to-treat estimate. - Audits assignment, baseline balance, sample-ratio mismatch, attrition, contamination, exposure, and spillovers without treating balance as the source of identification. - Uses appropriate user-level aggregation and uncertainty for the assignment design, repeated observations, and skewed revenue. - Uses pre-period outcomes, covariate adjustment, CUPED, or difference-in-differences only as prespecified precision improvements or sensitivity analyses. - Converts the estimated per-user lift into a scoped total without silently extrapolating beyond the supported population. - Discusses cannibalization, guardrails, data quality, and how conclusions would change if the identifying assumptions fail. ### Follow-up Questions 1. What if only 70 of the 100 assigned users actually see an ad? 2. How would you use pre-treatment revenue to improve precision without changing the primary estimand? 3. How would you distinguish ad revenue lift from cannibalization or a change in search activity? 4. How would you communicate a positive point estimate with a wide confidence interval?