Design an experiment for exploratory recommendations

Context

You are launching an online A/B test for a new version of a recommendation algorithm. The goal of the new algorithm is to increase users’ exploration behavior (discovering new or diverse content). A known challenge is that, in the short term, the new algorithm may slightly reduce core engagement metrics (e.g., like rate).

Tasks

1. Experiment design: How would you design and run this experiment to decide whether to launch the new algorithm?
2. Metric definition:
   • How would you define and measure “exploration” ?
   • How would you define and measure long-term user satisfaction ?
   • Besides short-term engagement (e.g., like rate), what long-term retention / ecosystem health metrics would you track?
3. Limited duration constraint: If the experiment window is limited and you cannot fully observe long-term impact, how would you analyze, interpret, and make a launch recommendation (e.g., ship / don’t ship / ramp gradually)?

Assumptions (you may state and adjust)

• Randomization unit can be user-level.
• You have event logs (impressions, clicks, likes, dwell time, follows, hides/blocks), content metadata (topic/category/creator), and user history.
• You can run holdouts or ramps if needed.