Design and analyze a banner A/B test
Overview
This question evaluates a candidate's competence in designing and analyzing A/B experiments, covering randomization and exposure decisions, precise metric definitions and guardrails, sample size and power calculations, analysis plans for multiple comparisons and covariate adjustment, and diagnostic validation within the Analytics & Experimentation domain. It is commonly asked because it tests both conceptual understanding and practical application of experimentation methodology—assessing the ability to manage measurement issues, statistical assumptions, and explicit decision rules that determine whether product changes are supported by the data.
A/B Test Design: Home-Page Banner
You are deciding whether to add a home-page banner in a consumer app. Design and analyze the A/B test end-to-end. Assume a typical logged-in user base with multiple sessions per user. Where needed, make minimal assumptions explicit so a first-time reader can follow.
- Randomization and exposure
- Choose the randomization unit (user-level vs session-level). Address cross-session consistency, eligibility/exposure, and potential interference.
- Metrics (primary and guardrails)
- Define: CTR, dwell time after click, retention, and revenue per session. Provide precise metric formulas and denominators.
- Precisely define accidental clicks and explain how to exclude or reweight them (e.g., dwell < 500 ms or immediate back within 2 s).
- Powering
- Given baseline CTR = 1.5% and expected relative lift = 10% (so 1.65% in treatment), compute per-arm sample size for 90% power and α = 0.05 (two-sided). Show formulas, assumptions (e.g., pooled variance), and any adjustments (e.g., trigger rate, clustering).
- Analysis plan
- Handle multiple banner placements (multiple comparisons), position bias, and novelty effects.
- Specify whether and how to use CUPED or pre-period covariate adjustment.
- Diagnostic checks
- Exposure logging validation, sample-ratio checks, bot filtering, and sequential monitoring with proper alpha spending.
- Decision rule
- State explicit ship/no-ship criteria and a fallback plan if accidental-click rates spike.
Solution
Login required