Design A/B Tests for Banner Ad and Group-Story Feature
Overview
This question evaluates a data scientist's skills in experimental design, metrics selection, causal inference, and interpretation of engagement and monetization signals for product changes like banner ads and group-story features.
Snapchat
Jul 12, 2025, 6:59 PM
Data Scientist
Onsite
Analytics & Experimentation
47
0
Product Decision Cases: Banner Ad and Group-Story Feature
Context
You are evaluating two product decisions in a consumer social app:
- Adding a new banner ad placement in the feed/stories surface.
- Launching a new group-story feature that lets multiple friends contribute to a shared story.
Assume you can run controlled experiments and have standard logging for impressions, clicks, views, sessions, and retention.
Questions
- Banner Ad
- What key metrics would you track to measure the impact of adding a banner ad versus not adding it?
- How would you design and analyze an A/B test to decide if the banner should be launched, including handling accidental clicks?
- Group-Story Feature
- Which success metrics would you define?
- How would you set up the experiment (units, sample size, duration) to decide on launch?
Hints: Consider CTR, dwell time, efficiency, retention, experiment design, guardrails, and segmentation.
Solution
Login required
Show