How to evaluate a new Carousel feature

## Context You are a Data Scientist at Pinterest. The product team wants to add a **horizontal Carousel** module at the **top of the Home feed** (similar to an Instagram Stories row). The Carousel shows a set of Pins (could be previously saved Pins, or recommended Pins). ## Task 1. **Clarify goals & use cases** - What questions would you ask product/engineering/design to clarify the purpose of the Carousel? - What hypotheses (user value + business value) would you write down? 2. **Define metrics** - Propose a measurement framework with: - **Primary metric(s)** (e.g., CTR definitions) - **Diagnostic metrics** (to understand behavior changes) - **Guardrail metrics** (to prevent harming the overall experience) - Be explicit about metric definitions (e.g., what counts as an impression/click for the Carousel vs the overall page). 3. **Experiment / evaluation plan** - Outline how you would evaluate the Carousel (e.g., A/B test design), including: - Unit of randomization - Key slices/segments - Ramp strategy / risk mitigation - Common pitfalls (novelty effects, logging changes, interference with the rest of the feed) 4. **Debugging: overall page CTR dropped after launch** Assume that after adding the Carousel, the **overall Home feed page CTR decreased**. - List plausible explanations (product + measurement + statistical). - Describe a step-by-step analysis plan to identify the root cause. - Explain how you’d decide whether the Carousel is actually harmful vs simply shifting clicks around (cannibalization) or changing denominators.