# Compare Shop and Web Ad Performance Without Overclaiming You have 28 days of daily ad data: ```text ads_detail(advertiser_id, ad_id, ad_type, ad_objective, ad_spend, ds) ads_convs(ad_id, num_conversions, ds) ``` An analyst asks, “Do Shop ads perform better than Web ads?” Describe how you would turn that question into a defensible analysis. Use only outcomes supported by these tables, show the core aggregation you would compute, and explain what you can and cannot conclude from an observational 28-day comparison. ### Clarifying Questions to Ask - Which business decision will this comparison inform? - Does “performance” mean conversions, conversions per dollar, total volume, or another outcome? - Are conversion counts attributed to the ad and date using a consistent window? - Can advertisers or objectives appear in both ad types, and how selected are they into each type? ### Part 1: Define the estimand and metrics Choose primary and supporting metrics that can be computed from the available spend and conversion fields. #### What This Part Should Cover - A precise unit of analysis and 28-day window - Weighted conversion efficiency rather than an average of unstable row ratios - Volume, spend, and data-quality context alongside the headline metric ### Part 2: Build the comparison Outline SQL or pseudocode that aggregates conversions and spend by ad type and supports useful stratified comparisons. #### What This Part Should Cover - Pre-aggregation of each table to a compatible ad-day grain - Protection against many-to-many joins - Segmentation by advertiser or objective where supported ### Part 3: Quantify uncertainty and limitations Explain how you would assess variability and why the result is not automatically a causal effect of ad type. #### What This Part Should Cover - Uncertainty that respects clustering and skew - Selection, objective mix, budget, and attribution differences - A path toward a randomized or stronger quasi-experimental design ### What a Strong Answer Covers - An operational definition of “better” tied to a decision - Correct aggregation and join grain - Heterogeneity and uncertainty, not just two point estimates - Clear separation of descriptive association from causal evidence ### Follow-up Questions 1. How would you compare the formats within advertisers that use both? 2. What if Shop ads have more conversions per dollar but far less total spend? 3. How would delayed conversions change the date logic? 4. What additional fields would you request before recommending a budget shift?