Design A/B Tests for Banner Ad and Group-Story Feature
Quick Overview
Evaluates A/B test design for a banner ad and a group-story feature in a social app. Strong answers define monetization, engagement, safety, and retention metrics, handle accidental clicks for ads, account for network effects in group stories, and set launch criteria with guardrails.
Design A/B Tests for Banner Ad and Group-Story Feature
Company: Snapchat
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
##### Scenario
Product decision cases:
1) whether to add a banner ad,
2) whether to launch a group-story feature.
##### Question
What key metrics would you track to measure the impact of adding a banner ad versus not adding it? How would you design and analyze an A/B test to decide if the banner should be launched, including handling accidental clicks? For the group-story feature, which success metrics would you define and how would you set up the experiment (units, sample size, duration) to decide on launch?
##### Hints
Think CTR, dwell time, efficiency, retention, experiment design, guardrails, segmentation.
Quick Answer: Evaluates A/B test design for a banner ad and a group-story feature in a social app. Strong answers define monetization, engagement, safety, and retention metrics, handle accidental clicks for ads, account for network effects in group stories, and set launch criteria with guardrails.
Design A/B Tests for a Banner Ad and a Group-Story Feature
You are evaluating two product decisions in a consumer social app: adding a new banner ad placement and launching a group-story feature that lets multiple friends contribute to a shared story.
Constraints & Assumptions
Treat these as two separate experiment-design cases.
Assume standard logging for impressions, clicks, story views, contributions, sessions, retention, revenue, and safety events.
Banner ads may affect monetization and user experience; group stories may create network effects.
Include accidental clicks, sample size, duration, randomization unit, and launch criteria where relevant.
Clarifying Questions to Ask Guidance
Where would the banner ad appear, and how often would users see it?
What counts as an intentional ad click?
How does the group-story feature work: who can create, invite, view, and contribute?
Are users connected in social groups that could create treatment spillovers?
Part 1 - Banner Ad Metrics
What key metrics would you track to measure the impact of adding a banner ad versus not adding it?
What This Part Should Cover Guidance
Monetization metrics such as impressions, CTR, quality clicks, revenue per user, RPM, fill rate, and advertiser outcomes.
User-experience guardrails such as session length, story/feed engagement, retention, hides, reports, complaints, accidental clicks, and app performance.
Segment analysis by new versus existing users, heavy versus light users, market, and placement.
Part 2 - Banner Ad Experiment
How would you design and analyze an A/B test to decide whether the banner ad should be launched, including handling accidental clicks?
What This Part Should Cover Guidance
User-level randomization, exposure logging, control and treatment definitions, sample size, duration, and ramp plan.
Quality-click definitions such as dwell time or landing-page engagement.
Guardrail thresholds and launch criteria that balance revenue lift against user harm.
Part 3 - Group-Story Metrics
Which success metrics would you define for the group-story feature?
Measures of meaningful group interaction, not just raw views.
Safety, spam, privacy, notification fatigue, and content-quality guardrails.
Part 4 - Group-Story Experiment
How would you set up the experiment, including units, sample size, and duration, to decide on launch?
What This Part Should Cover Guidance
Randomization choices such as user-level, cluster-level, or friend-graph/group-level assignment and how to handle spillovers.
Eligibility, exposure, power, duration, novelty effects, and heterogeneous treatment effects.
Launch criteria and monitoring for network effects and safety.
What a Strong Answer Covers Guidance
A strong answer separates the ad and social-feature cases, defines metrics that match each product goal, handles accidental clicks and network effects, and sets launch criteria using both primary metrics and guardrails.
Follow-up Questions Guidance
How would you estimate accidental click rate?
What if banner revenue rises but retention falls slightly?
How would you randomize group stories if friends invite each other across variants?