Design A/B Tests for a Banner Ad and a Group-Story Feature
You are evaluating two product decisions in a consumer social app: adding a new banner ad placement and launching a group-story feature that lets multiple friends contribute to a shared story.
Constraints & Assumptions
-
Treat these as two separate experiment-design cases.
-
Assume standard logging for impressions, clicks, story views, contributions, sessions, retention, revenue, and safety events.
-
Banner ads may affect monetization and user experience; group stories may create network effects.
-
Include accidental clicks, sample size, duration, randomization unit, and launch criteria where relevant.
Clarifying Questions to Ask
-
Where would the banner ad appear, and how often would users see it?
-
What counts as an intentional ad click?
-
How does the group-story feature work: who can create, invite, view, and contribute?
-
Are users connected in social groups that could create treatment spillovers?
Part 1 - Banner Ad Metrics
What key metrics would you track to measure the impact of adding a banner ad versus not adding it?
What This Part Should Cover
-
Monetization metrics such as impressions, CTR, quality clicks, revenue per user, RPM, fill rate, and advertiser outcomes.
-
User-experience guardrails such as session length, story/feed engagement, retention, hides, reports, complaints, accidental clicks, and app performance.
-
Segment analysis by new versus existing users, heavy versus light users, market, and placement.
Part 2 - Banner Ad Experiment
How would you design and analyze an A/B test to decide whether the banner ad should be launched, including handling accidental clicks?
What This Part Should Cover
-
User-level randomization, exposure logging, control and treatment definitions, sample size, duration, and ramp plan.
-
Quality-click definitions such as dwell time or landing-page engagement.
-
Guardrail thresholds and launch criteria that balance revenue lift against user harm.
Part 3 - Group-Story Metrics
Which success metrics would you define for the group-story feature?
What This Part Should Cover
-
Creation, invite, contribution, view, completion, reply, share, repeat usage, and retention metrics.
-
Measures of meaningful group interaction, not just raw views.
-
Safety, spam, privacy, notification fatigue, and content-quality guardrails.
Part 4 - Group-Story Experiment
How would you set up the experiment, including units, sample size, and duration, to decide on launch?
What This Part Should Cover
-
Randomization choices such as user-level, cluster-level, or friend-graph/group-level assignment and how to handle spillovers.
-
Eligibility, exposure, power, duration, novelty effects, and heterogeneous treatment effects.
-
Launch criteria and monitoring for network effects and safety.
What a Strong Answer Covers
A strong answer separates the ad and social-feature cases, defines metrics that match each product goal, handles accidental clicks and network effects, and sets launch criteria using both primary metrics and guardrails.
Follow-up Questions
-
How would you estimate accidental click rate?
-
What if banner revenue rises but retention falls slightly?
-
How would you randomize group stories if friends invite each other across variants?