Design an experiment for exploratory recommendations

# Experiment Design for Exploratory Recommendations You are launching an online A/B test for a new recommendation algorithm. The goal is to increase users' exploration behavior, meaning discovery of new or diverse content. A known challenge is that the new algorithm may slightly reduce core short-term engagement metrics such as like rate. Design the experiment, define exploration and long-term satisfaction metrics, and make a launch recommendation when the experiment window is too short to fully observe long-term impact. ### Constraints & Assumptions - Randomization can be user-level unless network effects require clustering. - Exploration should be valuable discovery, not random novelty. - Short-term engagement losses may be acceptable only if long-term value is credible. - Include retention, satisfaction, creator ecosystem, and safety guardrails. ### Clarifying Questions to Ask - What content universe and surfaces does the recommender affect? - What is the acceptable short-term engagement trade-off? - What long-term outcomes are historically linked to exploration? - Are survey, retention, and creator-side outcomes available? ### Part 1 - Experiment Design How would you design and run the experiment to decide whether to launch? #### What This Part Should Cover - Define treatment, control, eligibility, randomization unit, exposure, sample size, duration, and analysis plan. - Use user-level assignment with sticky variants; consider clusters if social spillovers are material. - Predefine decision rules balancing exploration lift and engagement guardrails. - Include SRM, logging checks, novelty, and segment analysis. ### Part 2 - Metric Definition How would you define exploration, long-term satisfaction, and ecosystem health? #### What This Part Should Cover - Exploration metrics may include content/category diversity, novelty, new creator discovery, entropy, distance from historical interests, and repeat engagement with discovered content. - Satisfaction metrics may include surveys, long-clicks, retention, return rate, session quality, negative feedback, and durable engagement. - Ecosystem metrics may include creator distribution, new creator exposure, safety, revenue, and content quality. ### Part 3 - Limited Duration Decision If the experiment window is limited and long-term impact is not fully observed, how would you analyze and recommend? #### What This Part Should Cover - Use validated leading indicators, proxy-to-long-term relationships, cohort early signals, and historical backtests. - Quantify uncertainty and downside risk. - Recommend ship, do not ship, ramp gradually, or extend test based on thresholds. - Use long-term holdouts or phased rollout to continue learning. ### Follow-up Questions - What if exploration increases but like rate drops? - How would you prevent the model from showing irrelevant content in the name of diversity? - How would you measure whether users return to newly discovered creators?