How to evaluate a new Carousel feature

Context

You are a Data Scientist at Pinterest. The product team wants to add a horizontal Carousel module at the top of the Home feed (similar to an Instagram Stories row). The Carousel shows a set of Pins (could be previously saved Pins, or recommended Pins).

Task

1. Clarify goals & use cases
   • What questions would you ask product/engineering/design to clarify the purpose of the Carousel?
   • What hypotheses (user value + business value) would you write down?
2. Define metrics
   • Propose a measurement framework with:
     • Primary metric(s) (e.g., CTR definitions)
     • Diagnostic metrics (to understand behavior changes)
     • Guardrail metrics (to prevent harming the overall experience)
   • Be explicit about metric definitions (e.g., what counts as an impression/click for the Carousel vs the overall page).
3. Experiment / evaluation plan
   • Outline how you would evaluate the Carousel (e.g., A/B test design), including:
     • Unit of randomization
     • Key slices/segments
     • Ramp strategy / risk mitigation
     • Common pitfalls (novelty effects, logging changes, interference with the rest of the feed)
4. Debugging: overall page CTR dropped after launch Assume that after adding the Carousel, the overall Home feed page CTR decreased .
   • List plausible explanations (product + measurement + statistical).
   • Describe a step-by-step analysis plan to identify the root cause.
   • Explain how you’d decide whether the Carousel is actually harmful vs simply shifting clicks around (cannibalization) or changing denominators.