How would you evaluate a carousel launch?
Quick Overview
This question evaluates a data scientist's skills in experimental design, product-metric specification, causal inference, and diagnostic analysis for online feature launches, with emphasis on trade-offs among engagement, retention, and revenue-related metrics.
You are the data scientist supporting Pinterest's home feed. Product wants to add a horizontally scrollable carousel at the top of the app, similar to Instagram Stories. The carousel could show saved Pins, recommended Pins, or another type of content. How would you evaluate whether this feature should launch?
In your answer, specify:
- What product goal(s) you would clarify first.
- The primary success metric(s) and guardrail metric(s). Consider trade-offs among page CTR, saves, session depth, retention, and ads or revenue.
- What experiment or observational analysis you would run, including randomization unit, exposure logging, and segmentation.
- How you would handle confounding such as novelty effects, selection bias, and changes in page layout.
Now assume an A/B test shows that overall home-page CTR decreased after the carousel was added. Explain the most plausible reasons this could happen and how you would diagnose whether the issue is metric definition, cannibalization of existing feed clicks, reduced below-the-fold exposure, latency, poor content relevance, or a logging bug. State how you would decide whether to launch, iterate, or stop.