Explain BLS vs CLS; compute t-stats
Overview
This question evaluates causal inference and experimental measurement skills in digital advertising, including distinctions between Brand Lift Studies and Conversion Lift Studies, sources of bias and variance, observational identification methods (Diff‑in‑Diff and Propensity Score Matching), and applied hypothesis testing for proportions and DiD variance estimation. It is commonly asked to gauge an interviewee's ability to reason about trade-offs between survey- and experiment-based metrics, state identification assumptions and robustness checks, and perform practical statistical inference; the domain is Statistics & Math for a data scientist role and the level spans both conceptual understanding and practical application.
Part A — Concepts: Define Brand Lift Study (BLS) vs Conversion Lift Study (CLS) in ads measurement. List key bias/variance sources for each (e.g., non-response bias in surveys, contamination, auction interference, selection). Describe when BLS and CLS might disagree and how you would reconcile them. Briefly contrast Diff-in-Diff (DiD) and Propensity Score Matching (PSM) for observational ads data; specify identification assumptions for each and one robustness check per method.
Part B — Hypothesis test for proportions (CLS): In a randomized CLS holdout, Treatment n1=1200 users, x1=42 conversions; Control n0=1180 users, x0=33 conversions. Compute: (1) point lift (p1−p0), (2) the pooled standard error under H0: p1=p0, and (3) the two-sided t/z-statistic and p-value. State whether the effect is significant at α=0.05.
Part C — Manual DiD t-stat (BLS survey): You measure Purchase Intent (binary) pre/post for exposed vs control cohorts: Group sizes: n_T_pre=300, n_T_post=320, n_C_pre=290, n_C_post=310. Sample means: m_T_pre=0.18, m_T_post=0.26, m_C_pre=0.17, m_C_post=0.20. Sample variances (Bernoulli sample variances): s2_T_pre=0.1476, s2_T_post=0.1924, s2_C_pre=0.1411, s2_C_post=0.1600. Tasks: (1) Compute the DiD estimator: (m_T_post−m_T_pre)−(m_C_post−m_C_pre). (2) Assuming independent samples and unequal group sizes, derive and compute an approximate standard error for the DiD using the sum of the four independent sample variances scaled by their n’s, and then (3) compute the t-statistic and conclude at α=0.05. Clearly show the formulas you use.